2024-09-16T07:07:17.877 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T07:07:17.897 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T07:07:17.985 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621 branch: reef description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} thrashers/sync-many workloads/pool-create-delete} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906621' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: reef ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - slow request - overall HEALTH_ sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 1390 sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 sleep_before_teardown: 0 subset: 5319/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 thrash_many: true - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T07:07:17.986 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa; will attempt to use it 2024-09-16T07:07:17.986 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa/tasks 2024-09-16T07:07:17.987 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T07:07:17.988 INFO:teuthology.task.internal:Checking packages... 2024-09-16T07:07:18.010 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8' 2024-09-16T07:07:18.010 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T07:07:18.010 INFO:teuthology.packaging:ref: None 2024-09-16T07:07:18.010 INFO:teuthology.packaging:tag: None 2024-09-16T07:07:18.010 INFO:teuthology.packaging:branch: reef 2024-09-16T07:07:18.010 INFO:teuthology.packaging:sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T07:07:18.010 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=reef 2024-09-16T07:07:18.266 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-853-gae7f3733-1focal 2024-09-16T07:07:18.268 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T07:07:18.274 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T07:07:18.275 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T07:07:18.314 INFO:teuthology.task.internal:Saving configuration 2024-09-16T07:07:18.329 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T07:07:18.339 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T07:07:18.368 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 07:01:31.513223', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T07:07:18.393 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 07:01:31.515264', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T07:07:18.394 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T07:07:18.402 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T07:07:18.402 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T07:07:18.402 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T07:07:18.488 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f44f9713130>, signals=[15]) 2024-09-16T07:07:18.488 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T07:07:18.494 INFO:teuthology.task.internal:Opening connections... 2024-09-16T07:07:18.495 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T07:07:18.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:07:18.586 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T07:07:18.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:07:18.675 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T07:07:18.683 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-09-16T07:07:18.690 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-09-16T07:07:18.690 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-09-16T07:07:18.737 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Ubuntu" 2024-09-16T07:07:18.737 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T07:07:18.738 INFO:teuthology.orchestra.run.smithi037.stdout:ID=ubuntu 2024-09-16T07:07:18.738 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE=debian 2024-09-16T07:07:18.738 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T07:07:18.738 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="20.04" 2024-09-16T07:07:18.738 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T07:07:18.738 INFO:teuthology.orchestra.run.smithi037.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T07:07:18.738 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T07:07:18.738 INFO:teuthology.orchestra.run.smithi037.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T07:07:18.738 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_CODENAME=focal 2024-09-16T07:07:18.738 INFO:teuthology.orchestra.run.smithi037.stdout:UBUNTU_CODENAME=focal 2024-09-16T07:07:18.739 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-09-16T07:07:18.768 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-09-16T07:07:18.775 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-09-16T07:07:18.776 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-09-16T07:07:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="Ubuntu" 2024-09-16T07:07:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T07:07:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:ID=ubuntu 2024-09-16T07:07:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE=debian 2024-09-16T07:07:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T07:07:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="20.04" 2024-09-16T07:07:18.822 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T07:07:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T07:07:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T07:07:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T07:07:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_CODENAME=focal 2024-09-16T07:07:18.823 INFO:teuthology.orchestra.run.smithi143.stdout:UBUNTU_CODENAME=focal 2024-09-16T07:07:18.823 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-09-16T07:07:18.852 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T07:07:18.863 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T07:07:18.881 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T07:07:18.881 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-09-16T07:07:18.884 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-09-16T07:07:18.889 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T07:07:18.922 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T07:07:18.922 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-09-16T07:07:18.925 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-09-16T07:07:18.937 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T07:07:18.999 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T07:07:19.015 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T07:07:19.015 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T07:07:19.015 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T07:07:19.016 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T07:07:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:07:19.021 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-09-16T07:07:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:07:19.022 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-09-16T07:07:19.028 INFO:teuthology.orchestra.run.smithi037.stdout:5.4.0-124-generic 2024-09-16T07:07:19.029 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.4.0-124-generic 2024-09-16T07:07:19.029 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get clean 2024-09-16T07:07:19.030 INFO:teuthology.orchestra.run.smithi143.stdout:5.4.0-124-generic 2024-09-16T07:07:19.030 INFO:teuthology.task.kernel:Running kernel on smithi143: 5.4.0-124-generic 2024-09-16T07:07:19.030 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get clean 2024-09-16T07:07:19.142 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-09-16T07:07:19.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2024-09-16T07:07:19.412 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T07:07:19.452 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T07:07:19.452 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T07:07:19.496 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T07:07:19.521 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T07:07:19.603 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T07:07:19.705 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T07:07:19.797 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T07:07:19.838 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T07:07:19.841 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T07:07:19.842 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T07:07:19.854 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T07:07:19.862 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T07:07:19.862 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T07:07:19.865 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T07:07:19.865 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T07:07:19.871 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T07:07:19.876 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T07:07:19.877 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T07:07:19.878 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T07:07:19.878 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T07:07:19.890 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T07:07:19.890 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T07:07:19.936 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T07:07:19.956 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T07:07:20.079 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T07:07:20.097 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-16T07:07:20.293 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T07:07:20.408 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-16T07:07:20.481 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T07:07:20.484 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T07:07:20.498 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T07:07:20.498 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T07:07:20.509 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T07:07:20.522 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T07:07:20.523 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T07:07:20.523 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-16T07:07:20.588 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T07:07:20.590 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T07:07:20.592 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T07:07:20.601 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T07:07:20.602 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T07:07:20.604 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T07:07:20.604 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T07:07:20.605 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-16T07:07:20.617 INFO:teuthology.orchestra.run.smithi143.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-16T07:07:20.621 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T07:07:20.633 INFO:teuthology.orchestra.run.smithi143.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T07:07:20.633 INFO:teuthology.orchestra.run.smithi143.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T07:07:20.652 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T07:07:20.659 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T07:07:20.659 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T07:07:20.659 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T07:07:20.670 INFO:teuthology.orchestra.run.smithi143.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T07:07:20.689 INFO:teuthology.orchestra.run.smithi143.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T07:07:20.698 INFO:teuthology.orchestra.run.smithi143.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T07:07:20.698 INFO:teuthology.orchestra.run.smithi143.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T07:07:20.699 INFO:teuthology.orchestra.run.smithi143.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T07:07:20.699 INFO:teuthology.orchestra.run.smithi143.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T07:07:20.700 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-16T07:07:20.711 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T07:07:20.711 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T07:07:20.727 INFO:teuthology.orchestra.run.smithi143.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T07:07:20.727 INFO:teuthology.orchestra.run.smithi143.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T07:07:20.728 INFO:teuthology.orchestra.run.smithi143.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T07:07:20.731 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T07:07:20.731 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T07:07:20.741 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T07:07:20.774 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T07:07:20.780 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T07:07:20.780 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T07:07:20.781 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T07:07:20.782 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T07:07:20.808 INFO:teuthology.orchestra.run.smithi143.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T07:07:20.809 INFO:teuthology.orchestra.run.smithi143.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T07:07:20.809 INFO:teuthology.orchestra.run.smithi143.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T07:07:20.810 INFO:teuthology.orchestra.run.smithi143.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T07:07:20.835 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T07:07:20.835 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T07:07:20.836 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T07:07:20.837 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T07:07:20.917 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T07:07:20.917 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T07:07:20.918 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T07:07:24.316 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 21.8 MB in 3s (7,430 kB/s) 2024-09-16T07:07:24.919 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 21.8 MB in 4s (5,956 kB/s) 2024-09-16T07:07:25.862 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T07:07:25.890 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T07:07:25.963 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T07:07:26.181 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2024-09-16T07:07:26.181 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2024-09-16T07:07:26.390 INFO:teuthology.orchestra.run.smithi143.stdout:The following additional packages will be installed: 2024-09-16T07:07:26.390 INFO:teuthology.orchestra.run.smithi143.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T07:07:26.390 INFO:teuthology.orchestra.run.smithi143.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T07:07:26.390 INFO:teuthology.orchestra.run.smithi143.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T07:07:26.391 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2024-09-16T07:07:26.391 INFO:teuthology.orchestra.run.smithi143.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T07:07:26.391 INFO:teuthology.orchestra.run.smithi143.stdout:Recommended packages: 2024-09-16T07:07:26.391 INFO:teuthology.orchestra.run.smithi143.stdout: thermald 2024-09-16T07:07:26.448 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2024-09-16T07:07:26.448 INFO:teuthology.orchestra.run.smithi143.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T07:07:26.449 INFO:teuthology.orchestra.run.smithi143.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T07:07:26.449 INFO:teuthology.orchestra.run.smithi143.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T07:07:26.450 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages will be upgraded: 2024-09-16T07:07:26.451 INFO:teuthology.orchestra.run.smithi143.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T07:07:26.503 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T07:07:26.532 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T07:07:26.626 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T07:07:26.633 INFO:teuthology.orchestra.run.smithi143.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T07:07:26.633 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 77.2 MB of archives. 2024-09-16T07:07:26.633 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T07:07:26.633 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T07:07:26.849 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-09-16T07:07:26.850 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-09-16T07:07:27.063 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2024-09-16T07:07:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T07:07:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T07:07:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T07:07:27.065 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2024-09-16T07:07:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T07:07:27.065 INFO:teuthology.orchestra.run.smithi037.stdout:Recommended packages: 2024-09-16T07:07:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: thermald 2024-09-16T07:07:27.123 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2024-09-16T07:07:27.124 INFO:teuthology.orchestra.run.smithi037.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T07:07:27.124 INFO:teuthology.orchestra.run.smithi037.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T07:07:27.124 INFO:teuthology.orchestra.run.smithi037.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T07:07:27.125 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages will be upgraded: 2024-09-16T07:07:27.126 INFO:teuthology.orchestra.run.smithi037.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T07:07:27.307 INFO:teuthology.orchestra.run.smithi037.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T07:07:27.307 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 77.2 MB of archives. 2024-09-16T07:07:27.307 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T07:07:27.307 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T07:07:27.792 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T07:07:28.065 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T07:07:28.452 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T07:07:28.724 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T07:07:29.398 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T07:07:29.399 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T07:07:29.399 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T07:07:29.680 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T07:07:29.706 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T07:07:30.050 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T07:07:30.050 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T07:07:30.050 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T07:07:30.332 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T07:07:30.371 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T07:07:30.413 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 77.2 MB in 3s (23.8 MB/s) 2024-09-16T07:07:30.598 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T07:07:31.150 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 77.2 MB in 3s (23.8 MB/s) 2024-09-16T07:07:31.308 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T07:07:32.270 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T07:07:32.275 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:07:32.336 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:33.010 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T07:07:33.015 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:07:33.105 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:34.268 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T07:07:34.282 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:07:34.378 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:34.964 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T07:07:34.977 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:07:35.073 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:35.430 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T07:07:35.444 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:07:35.488 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:36.200 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T07:07:36.213 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:07:36.250 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:40.071 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T07:07:40.197 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T07:07:40.532 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T07:07:40.658 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T07:07:41.011 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T07:07:41.026 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T07:07:41.069 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T07:07:41.130 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T07:07:41.256 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T07:07:41.599 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T07:07:41.726 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T07:07:42.054 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T07:07:42.069 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T07:07:42.104 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T07:07:46.344 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T07:07:46.367 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:07:46.410 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:47.914 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T07:07:47.936 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T07:07:47.972 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:49.066 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T07:07:49.187 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T07:07:49.515 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:50.736 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:50.818 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T07:07:50.939 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T07:07:51.298 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:51.884 INFO:teuthology.orchestra.run.smithi143.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T07:07:51.884 INFO:teuthology.orchestra.run.smithi143.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T07:07:52.013 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:52.586 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:53.206 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T07:07:53.332 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T07:07:53.458 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:53.593 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T07:07:53.736 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T07:07:53.862 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:53.935 INFO:teuthology.orchestra.run.smithi143.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T07:07:53.935 INFO:teuthology.orchestra.run.smithi143.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T07:07:54.279 INFO:teuthology.orchestra.run.smithi037.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T07:07:54.279 INFO:teuthology.orchestra.run.smithi037.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T07:07:54.722 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:55.956 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T07:07:56.084 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T07:07:56.209 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:56.335 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T07:07:56.486 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T07:07:56.612 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T07:07:56.691 INFO:teuthology.orchestra.run.smithi037.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T07:07:56.691 INFO:teuthology.orchestra.run.smithi037.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T07:08:16.291 INFO:teuthology.orchestra.run.smithi143.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T07:08:16.291 INFO:teuthology.orchestra.run.smithi143.stdout:Sourcing file `/etc/default/grub' 2024-09-16T07:08:16.299 INFO:teuthology.orchestra.run.smithi143.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T07:08:16.322 INFO:teuthology.orchestra.run.smithi143.stdout:Generating grub configuration file ... 2024-09-16T07:08:16.622 INFO:teuthology.orchestra.run.smithi143.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T07:08:16.642 INFO:teuthology.orchestra.run.smithi143.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T07:08:17.038 INFO:teuthology.orchestra.run.smithi143.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T07:08:17.049 INFO:teuthology.orchestra.run.smithi143.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T07:08:19.057 INFO:teuthology.orchestra.run.smithi037.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T07:08:19.057 INFO:teuthology.orchestra.run.smithi037.stdout:Sourcing file `/etc/default/grub' 2024-09-16T07:08:19.077 INFO:teuthology.orchestra.run.smithi037.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T07:08:19.112 INFO:teuthology.orchestra.run.smithi037.stdout:Generating grub configuration file ... 2024-09-16T07:08:19.414 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T07:08:19.445 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T07:08:19.517 INFO:teuthology.orchestra.run.smithi143.stdout:done 2024-09-16T07:08:19.829 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T07:08:19.841 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T07:08:19.911 DEBUG:teuthology.orchestra.run.smithi143:> dpkg -s linux-image-generic 2024-09-16T07:08:19.965 INFO:teuthology.orchestra.run.smithi143.stdout:Package: linux-image-generic 2024-09-16T07:08:19.965 INFO:teuthology.orchestra.run.smithi143.stdout:Status: install ok installed 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Priority: optional 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Section: kernel 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Installed-Size: 20 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: amd64 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Source: linux-meta 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Version: 5.4.0.195.193 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Recommends: thermald 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout:Description: Generic Linux kernel image 2024-09-16T07:08:19.966 INFO:teuthology.orchestra.run.smithi143.stdout: This package will always depend on the latest generic kernel image 2024-09-16T07:08:19.967 INFO:teuthology.orchestra.run.smithi143.stdout: available. 2024-09-16T07:08:19.967 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T07:08:19.967 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T07:08:19.967 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T07:08:19.967 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T07:08:19.967 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi143.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T07:08:19.968 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get clean 2024-09-16T07:08:20.052 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2024-09-16T07:08:20.185 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T07:08:20.301 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T07:08:20.383 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T07:08:20.467 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T07:08:22.034 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T07:08:22.059 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T07:08:22.133 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T07:08:22.350 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2024-09-16T07:08:22.351 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2024-09-16T07:08:22.371 INFO:teuthology.orchestra.run.smithi037.stdout:done 2024-09-16T07:08:22.594 INFO:teuthology.orchestra.run.smithi143.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T07:08:22.595 INFO:teuthology.orchestra.run.smithi143.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T07:08:22.596 DEBUG:teuthology.orchestra.run.smithi143:> dpkg -s linux-image-generic 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Package: linux-image-generic 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Status: install ok installed 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Priority: optional 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Section: kernel 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Installed-Size: 20 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: amd64 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Source: linux-meta 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Version: 5.4.0.195.193 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T07:08:22.615 INFO:teuthology.orchestra.run.smithi143.stdout:Recommends: thermald 2024-09-16T07:08:22.616 INFO:teuthology.orchestra.run.smithi143.stdout:Description: Generic Linux kernel image 2024-09-16T07:08:22.616 INFO:teuthology.orchestra.run.smithi143.stdout: This package will always depend on the latest generic kernel image 2024-09-16T07:08:22.616 INFO:teuthology.orchestra.run.smithi143.stdout: available. 2024-09-16T07:08:22.616 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-09-16T07:08:22.620 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.9BbpKdh0Ni 2024-09-16T07:08:22.621 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /boot/grub/grub.cfg /tmp/tmp.9BbpKdh0Ni 2024-09-16T07:08:22.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.9BbpKdh0Ni 2024-09-16T07:08:22.718 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.9BbpKdh0Ni is 9KB 2024-09-16T07:08:22.733 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.9BbpKdh0Ni 2024-09-16T07:08:22.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T07:08:22.795 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:08:22.795 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T07:08:22.795 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T07:08:22.825 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T07:08:22.825 DEBUG:teuthology.orchestra.run.smithi143:> sudo update-grub 2024-09-16T07:08:22.850 DEBUG:teuthology.orchestra.run.smithi037:> dpkg -s linux-image-generic 2024-09-16T07:08:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:Package: linux-image-generic 2024-09-16T07:08:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:Status: install ok installed 2024-09-16T07:08:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:Priority: optional 2024-09-16T07:08:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:Section: kernel 2024-09-16T07:08:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:Installed-Size: 20 2024-09-16T07:08:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T07:08:22.897 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: amd64 2024-09-16T07:08:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:Source: linux-meta 2024-09-16T07:08:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:Version: 5.4.0.195.193 2024-09-16T07:08:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T07:08:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T07:08:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:Recommends: thermald 2024-09-16T07:08:22.898 INFO:teuthology.orchestra.run.smithi037.stdout:Description: Generic Linux kernel image 2024-09-16T07:08:22.898 INFO:teuthology.orchestra.run.smithi037.stdout: This package will always depend on the latest generic kernel image 2024-09-16T07:08:22.898 INFO:teuthology.orchestra.run.smithi037.stdout: available. 2024-09-16T07:08:22.898 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T07:08:22.898 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T07:08:22.898 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T07:08:22.899 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T07:08:22.899 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T07:08:22.899 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get clean 2024-09-16T07:08:22.969 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-09-16T07:08:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T07:08:23.232 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T07:08:23.302 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T07:08:23.392 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T07:08:24.138 INFO:teuthology.orchestra.run.smithi143.stderr:Sourcing file `/etc/default/grub' 2024-09-16T07:08:24.144 INFO:teuthology.orchestra.run.smithi143.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T07:08:24.154 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2024-09-16T07:08:24.404 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T07:08:24.415 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T07:08:24.770 INFO:teuthology.orchestra.run.smithi143.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T07:08:24.781 INFO:teuthology.orchestra.run.smithi143.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T07:08:25.142 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T07:08:25.170 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T07:08:25.247 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T07:08:25.469 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-09-16T07:08:25.470 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-09-16T07:08:25.713 INFO:teuthology.orchestra.run.smithi037.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T07:08:25.713 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T07:08:25.715 DEBUG:teuthology.orchestra.run.smithi037:> dpkg -s linux-image-generic 2024-09-16T07:08:25.736 INFO:teuthology.orchestra.run.smithi037.stdout:Package: linux-image-generic 2024-09-16T07:08:25.736 INFO:teuthology.orchestra.run.smithi037.stdout:Status: install ok installed 2024-09-16T07:08:25.736 INFO:teuthology.orchestra.run.smithi037.stdout:Priority: optional 2024-09-16T07:08:25.736 INFO:teuthology.orchestra.run.smithi037.stdout:Section: kernel 2024-09-16T07:08:25.736 INFO:teuthology.orchestra.run.smithi037.stdout:Installed-Size: 20 2024-09-16T07:08:25.736 INFO:teuthology.orchestra.run.smithi037.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T07:08:25.736 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: amd64 2024-09-16T07:08:25.736 INFO:teuthology.orchestra.run.smithi037.stdout:Source: linux-meta 2024-09-16T07:08:25.736 INFO:teuthology.orchestra.run.smithi037.stdout:Version: 5.4.0.195.193 2024-09-16T07:08:25.737 INFO:teuthology.orchestra.run.smithi037.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T07:08:25.737 INFO:teuthology.orchestra.run.smithi037.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T07:08:25.737 INFO:teuthology.orchestra.run.smithi037.stdout:Recommends: thermald 2024-09-16T07:08:25.737 INFO:teuthology.orchestra.run.smithi037.stdout:Description: Generic Linux kernel image 2024-09-16T07:08:25.737 INFO:teuthology.orchestra.run.smithi037.stdout: This package will always depend on the latest generic kernel image 2024-09-16T07:08:25.737 INFO:teuthology.orchestra.run.smithi037.stdout: available. 2024-09-16T07:08:25.737 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-09-16T07:08:25.741 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.onAeDxh3VK 2024-09-16T07:08:25.741 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub/grub.cfg /tmp/tmp.onAeDxh3VK 2024-09-16T07:08:25.810 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.onAeDxh3VK 2024-09-16T07:08:25.852 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.onAeDxh3VK is 9KB 2024-09-16T07:08:25.866 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.onAeDxh3VK 2024-09-16T07:08:25.873 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T07:08:25.931 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:08:25.931 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T07:08:25.931 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T07:08:25.967 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T07:08:25.968 DEBUG:teuthology.orchestra.run.smithi037:> sudo update-grub 2024-09-16T07:08:26.191 INFO:teuthology.orchestra.run.smithi143.stderr:done 2024-09-16T07:08:26.193 DEBUG:teuthology.orchestra.run.smithi143:> sudo shutdown -r now 2024-09-16T07:08:27.322 INFO:teuthology.orchestra.run.smithi037.stderr:Sourcing file `/etc/default/grub' 2024-09-16T07:08:27.334 INFO:teuthology.orchestra.run.smithi037.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T07:08:27.344 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-09-16T07:08:27.632 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T07:08:27.643 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T07:08:27.984 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T07:08:27.996 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T07:08:29.452 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-09-16T07:08:29.453 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-09-16T07:08:56.221 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T07:08:56.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T07:08:56.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:08:59.459 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T07:08:59.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T07:08:59.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:09:18.017 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-09-16T07:09:27.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T07:09:27.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:09:45.405 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-09-16T07:09:56.233 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T07:09:57.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T07:09:57.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:10:00.255 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-09-16T07:10:05.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T07:10:05.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:10:05.684 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-09-16T07:10:06.776 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T07:10:06.776 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-195-generic"... 2024-09-16T07:10:06.776 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-09-16T07:10:06.782 INFO:teuthology.orchestra.run.smithi143.stdout:5.4.0-195-generic 2024-09-16T07:10:06.783 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T07:10:06.783 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T07:10:06.783 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T07:10:07.783 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T07:10:07.784 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T07:10:07.855 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2024-09-16T07:10:07.886 DEBUG:teuthology.parallel:result is None 2024-09-16T07:10:15.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T07:10:15.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:10:15.508 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-09-16T07:10:16.081 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T07:10:16.084 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-16T07:10:16.085 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-09-16T07:10:16.091 INFO:teuthology.orchestra.run.smithi037.stdout:5.4.0-195-generic 2024-09-16T07:10:16.091 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T07:10:16.091 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T07:10:16.091 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T07:10:17.092 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T07:10:17.093 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T07:10:17.179 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-09-16T07:10:17.200 DEBUG:teuthology.parallel:result is None 2024-09-16T07:10:17.200 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T07:10:17.212 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T07:10:17.212 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T07:10:17.214 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T07:10:17.221 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T07:10:17.260 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T07:10:17.291 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T07:10:17.292 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T07:10:17.294 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T07:10:17.320 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T07:10:17.328 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T07:10:17.328 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T07:10:17.347 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T07:10:17.379 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:10:17.380 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:10:17.390 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:10:17.391 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T07:10:17.393 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T07:10:17.409 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T07:10:17.409 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T07:10:17.411 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T07:10:17.426 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T07:10:17.443 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T07:10:17.443 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T07:10:17.445 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T07:10:17.451 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T07:10:17.496 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T07:10:17.503 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:10:17.503 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T07:10:17.527 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T07:10:17.534 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T07:10:17.541 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:10:17.541 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T07:10:17.606 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-09-16T07:10:17.608 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-09-16T07:10:17.654 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T07:10:17.660 INFO:teuthology.task.internal:Starting timer... 2024-09-16T07:10:17.661 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T07:10:17.704 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T07:10:17.761 DEBUG:teuthology.task.selinux:Excluding smithi037: OS 'ubuntu' does not support SELinux 2024-09-16T07:10:17.761 DEBUG:teuthology.task.selinux:Excluding smithi143: OS 'ubuntu' does not support SELinux 2024-09-16T07:10:17.761 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T07:10:17.761 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T07:10:17.761 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T07:10:17.762 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T07:10:17.800 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T07:10:17.813 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T07:10:17.814 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com,smithi143.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T07:15:58.108 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi143.front.sepia.ceph.com')] 2024-09-16T07:15:58.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T07:15:58.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:15:58.201 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-09-16T07:15:58.322 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-16T07:15:58.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T07:15:58.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T07:15:58.417 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-09-16T07:15:58.542 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-16T07:15:58.542 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T07:15:58.552 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T07:15:58.552 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T07:15:58.553 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T07:15:58.555 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T07:15:58.555 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T07:15:58.588 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T07:15:58.589 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: Command line: ntpd -gq 2024-09-16T07:15:58.591 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: proto: precision = 0.095 usec (-23) 2024-09-16T07:15:58.592 INFO:teuthology.orchestra.run.smithi037.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T07:15:58.593 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T07:15:58.593 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: Command line: ntpd -gq 2024-09-16T07:15:58.593 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: proto: precision = 0.100 usec (-23) 2024-09-16T07:15:58.593 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T07:15:58.593 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T07:15:58.594 INFO:teuthology.orchestra.run.smithi143.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T07:15:58.594 INFO:teuthology.orchestra.run.smithi037.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T07:15:58.594 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T07:15:58.595 INFO:teuthology.orchestra.run.smithi143.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T07:15:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T07:15:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T07:15:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T07:15:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: Listen normally on 3 ens1f0 172.21.15.37:123 2024-09-16T07:15:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: Listen normally on 4 lo [::1]:123 2024-09-16T07:15:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe88:7794%4]:123 2024-09-16T07:15:58.596 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:15:58 ntpd[13659]: Listening on routing socket on fd #22 for interface updates 2024-09-16T07:15:58.597 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T07:15:58.597 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T07:15:58.597 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T07:15:58.597 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T07:15:58.597 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: Listen normally on 3 enp3s0f1 172.21.15.143:123 2024-09-16T07:15:58.597 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: Listen normally on 4 lo [::1]:123 2024-09-16T07:15:58.597 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7783%5]:123 2024-09-16T07:15:58.597 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:15:58 ntpd[13731]: Listening on routing socket on fd #22 for interface updates 2024-09-16T07:16:00.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:00 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:00.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:00 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:00.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:00 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:00.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:00 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:00.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:00 ntpd[13659]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:00.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:00 ntpd[13659]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:00.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:00.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:00 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:01 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:01 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:02.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:02 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:02.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:02 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:02.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:02 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:02.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:02 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:02.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:02 ntpd[13659]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:02.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:02 ntpd[13659]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:02.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:02.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:02.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:02.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:02.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:02.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:02.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:02.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:02.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:02.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:02.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:02.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:02 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:03 ntpd[13659]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:03.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:03 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:03.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:03 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:03.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:03 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:03.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:03 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:03.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:03 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:03.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:03 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:03.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:03 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:03.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:03 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:04 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:04.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:04 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:04.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:04 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:04.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:04 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:04.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:04 ntpd[13659]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:04.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:04 ntpd[13659]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:04.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:04.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:04.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:04.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:04.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:04.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:04.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:04.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:04.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:04.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:04.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:04.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:04 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:05 ntpd[13659]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:05.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:05 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:05.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:05 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:05.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:05 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:05.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:05 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:05.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:05 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:05.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:05 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:06.589 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:06 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:06.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:06 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:06.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:06 ntpd[13659]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:06.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:06 ntpd[13659]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:06.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:06 ntpd[13659]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:06.590 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:06 ntpd[13659]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:06.590 INFO:teuthology.orchestra.run.smithi037.stdout:16 Sep 07:16:06 ntpd[13659]: ntpd: time slew +0.007820 s 2024-09-16T07:16:06.590 INFO:teuthology.orchestra.run.smithi037.stdout:ntpd: time slew +0.007820s 2024-09-16T07:16:06.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:06 ntpd[13659]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T07:16:06.591 INFO:teuthology.orchestra.run.smithi037.stderr:16 Sep 07:16:06 ntpd[13659]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T07:16:06.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:06 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:06.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:06 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:06.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:06 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:06.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:06 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:06.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:06 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T07:16:06.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:06 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T07:16:06.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:06 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T07:16:06.592 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:06 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T07:16:06.593 INFO:teuthology.orchestra.run.smithi143.stdout:16 Sep 07:16:06 ntpd[13731]: ntpd: time slew +0.000058 s 2024-09-16T07:16:06.593 INFO:teuthology.orchestra.run.smithi143.stdout:ntpd: time slew +0.000058s 2024-09-16T07:16:06.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:06 ntpd[13731]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T07:16:06.593 INFO:teuthology.orchestra.run.smithi143.stderr:16 Sep 07:16:06 ntpd[13731]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T07:16:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T07:16:06.654 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================== 2024-09-16T07:16:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:16:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:16:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:16:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:16:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T07:16:06.673 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2024-09-16T07:16:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:16:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:16:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:16:06.673 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:16:06.674 INFO:teuthology.run_tasks:Running task install... 2024-09-16T07:16:06.703 DEBUG:teuthology.task.install:project ceph 2024-09-16T07:16:06.703 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}} 2024-09-16T07:16:06.703 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'} 2024-09-16T07:16:06.703 INFO:teuthology.task.install:Using flavor: default 2024-09-16T07:16:06.709 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T07:16:06.709 INFO:teuthology.task.install:extra packages: [] 2024-09-16T07:16:06.710 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-key list | grep Ceph 2024-09-16T07:16:06.710 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-key list | grep Ceph 2024-09-16T07:16:06.813 INFO:teuthology.orchestra.run.smithi037.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T07:16:06.817 INFO:teuthology.orchestra.run.smithi143.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T07:16:06.870 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T07:16:06.871 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T07:16:06.872 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T07:16:06.872 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T07:16:06.877 INFO:teuthology.orchestra.run.smithi037.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T07:16:06.877 INFO:teuthology.orchestra.run.smithi037.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T07:16:06.878 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T07:16:06.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T07:16:07.027 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default/ 2024-09-16T07:16:07.027 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1focal 2024-09-16T07:16:07.050 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default/ 2024-09-16T07:16:07.050 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1focal 2024-09-16T07:16:07.142 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:16:07.142 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T07:16:07.158 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-09-16T07:16:07.163 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:16:07.163 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T07:16:07.179 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2024-09-16T07:16:07.317 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T07:16:07.325 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T07:16:07.342 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T07:16:07.348 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T07:16:07.355 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T07:16:07.370 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal InRelease 2024-09-16T07:16:07.372 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T07:16:07.392 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T07:16:07.402 INFO:teuthology.orchestra.run.smithi037.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal InRelease 2024-09-16T07:16:07.414 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T07:16:07.423 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T07:16:07.434 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T07:16:07.436 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main i386 Packages [3,420 B] 2024-09-16T07:16:07.446 INFO:teuthology.orchestra.run.smithi037.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T07:16:07.465 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T07:16:07.468 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-16T07:16:07.468 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-16T07:16:07.501 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main i386 Packages [3,420 B] 2024-09-16T07:16:07.968 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 28.7 kB in 1s (45.9 kB/s) 2024-09-16T07:16:08.014 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 28.7 kB in 1s (44.9 kB/s) 2024-09-16T07:16:09.128 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T07:16:09.155 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1focal cephadm=18.2.4-845-ge4e227b3-1focal ceph-mds=18.2.4-845-ge4e227b3-1focal ceph-mgr=18.2.4-845-ge4e227b3-1focal ceph-common=18.2.4-845-ge4e227b3-1focal ceph-fuse=18.2.4-845-ge4e227b3-1focal ceph-test=18.2.4-845-ge4e227b3-1focal ceph-volume=18.2.4-845-ge4e227b3-1focal radosgw=18.2.4-845-ge4e227b3-1focal python3-rados=18.2.4-845-ge4e227b3-1focal python3-rgw=18.2.4-845-ge4e227b3-1focal python3-cephfs=18.2.4-845-ge4e227b3-1focal python3-rbd=18.2.4-845-ge4e227b3-1focal libcephfs2=18.2.4-845-ge4e227b3-1focal libcephfs-dev=18.2.4-845-ge4e227b3-1focal librados2=18.2.4-845-ge4e227b3-1focal librbd1=18.2.4-845-ge4e227b3-1focal rbd-fuse=18.2.4-845-ge4e227b3-1focal 2024-09-16T07:16:09.213 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T07:16:09.226 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2024-09-16T07:16:09.240 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1focal cephadm=18.2.4-845-ge4e227b3-1focal ceph-mds=18.2.4-845-ge4e227b3-1focal ceph-mgr=18.2.4-845-ge4e227b3-1focal ceph-common=18.2.4-845-ge4e227b3-1focal ceph-fuse=18.2.4-845-ge4e227b3-1focal ceph-test=18.2.4-845-ge4e227b3-1focal ceph-volume=18.2.4-845-ge4e227b3-1focal radosgw=18.2.4-845-ge4e227b3-1focal python3-rados=18.2.4-845-ge4e227b3-1focal python3-rgw=18.2.4-845-ge4e227b3-1focal python3-cephfs=18.2.4-845-ge4e227b3-1focal python3-rbd=18.2.4-845-ge4e227b3-1focal libcephfs2=18.2.4-845-ge4e227b3-1focal libcephfs-dev=18.2.4-845-ge4e227b3-1focal librados2=18.2.4-845-ge4e227b3-1focal librbd1=18.2.4-845-ge4e227b3-1focal rbd-fuse=18.2.4-845-ge4e227b3-1focal 2024-09-16T07:16:09.314 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-09-16T07:16:09.443 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2024-09-16T07:16:09.444 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2024-09-16T07:16:09.514 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-09-16T07:16:09.515 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-09-16T07:16:09.646 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T07:16:09.647 INFO:teuthology.orchestra.run.smithi143.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T07:16:09.648 INFO:teuthology.orchestra.run.smithi143.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T07:16:09.648 INFO:teuthology.orchestra.run.smithi143.stdout:The following additional packages will be installed: 2024-09-16T07:16:09.648 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T07:16:09.648 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T07:16:09.648 INFO:teuthology.orchestra.run.smithi143.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T07:16:09.648 INFO:teuthology.orchestra.run.smithi143.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T07:16:09.648 INFO:teuthology.orchestra.run.smithi143.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T07:16:09.649 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T07:16:09.650 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T07:16:09.651 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T07:16:09.651 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T07:16:09.651 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T07:16:09.651 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T07:16:09.651 INFO:teuthology.orchestra.run.smithi143.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-16T07:16:09.651 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout:Recommended packages: 2024-09-16T07:16:09.652 INFO:teuthology.orchestra.run.smithi143.stdout: btrfs-tools 2024-09-16T07:16:09.720 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T07:16:09.721 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T07:16:09.721 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T07:16:09.721 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2024-09-16T07:16:09.721 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T07:16:09.722 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T07:16:09.722 INFO:teuthology.orchestra.run.smithi037.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T07:16:09.722 INFO:teuthology.orchestra.run.smithi037.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T07:16:09.722 INFO:teuthology.orchestra.run.smithi037.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T07:16:09.723 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T07:16:09.723 INFO:teuthology.orchestra.run.smithi037.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-16T07:16:09.723 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T07:16:09.723 INFO:teuthology.orchestra.run.smithi037.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T07:16:09.723 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2024-09-16T07:16:09.723 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2024-09-16T07:16:09.723 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T07:16:09.723 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T07:16:09.723 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T07:16:09.724 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T07:16:09.724 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T07:16:09.724 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T07:16:09.724 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T07:16:09.724 INFO:teuthology.orchestra.run.smithi037.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T07:16:09.724 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T07:16:09.724 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T07:16:09.724 INFO:teuthology.orchestra.run.smithi037.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-16T07:16:09.724 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2024-09-16T07:16:09.724 INFO:teuthology.orchestra.run.smithi037.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T07:16:09.725 INFO:teuthology.orchestra.run.smithi037.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T07:16:09.725 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T07:16:09.725 INFO:teuthology.orchestra.run.smithi037.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T07:16:09.725 INFO:teuthology.orchestra.run.smithi037.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T07:16:09.725 INFO:teuthology.orchestra.run.smithi037.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T07:16:09.725 INFO:teuthology.orchestra.run.smithi037.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T07:16:09.725 INFO:teuthology.orchestra.run.smithi037.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T07:16:09.725 INFO:teuthology.orchestra.run.smithi037.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T07:16:09.725 INFO:teuthology.orchestra.run.smithi037.stdout:Recommended packages: 2024-09-16T07:16:09.725 INFO:teuthology.orchestra.run.smithi037.stdout: btrfs-tools 2024-09-16T07:16:09.772 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2024-09-16T07:16:09.772 INFO:teuthology.orchestra.run.smithi143.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T07:16:09.772 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T07:16:09.772 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T07:16:09.772 INFO:teuthology.orchestra.run.smithi143.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T07:16:09.772 INFO:teuthology.orchestra.run.smithi143.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T07:16:09.773 INFO:teuthology.orchestra.run.smithi143.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T07:16:09.773 INFO:teuthology.orchestra.run.smithi143.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T07:16:09.774 INFO:teuthology.orchestra.run.smithi143.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-16T07:16:09.774 INFO:teuthology.orchestra.run.smithi143.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-16T07:16:09.774 INFO:teuthology.orchestra.run.smithi143.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-16T07:16:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2024-09-16T07:16:09.776 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet 2024-09-16T07:16:09.776 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages will be upgraded: 2024-09-16T07:16:09.776 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 librbd1 2024-09-16T07:16:09.840 INFO:teuthology.orchestra.run.smithi143.stdout:2 upgraded, 110 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T07:16:09.840 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 199 MB of archives. 2024-09-16T07:16:09.840 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 977 MB of additional disk space will be used. 2024-09-16T07:16:09.840 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T07:16:09.849 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2024-09-16T07:16:09.849 INFO:teuthology.orchestra.run.smithi037.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T07:16:09.849 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T07:16:09.849 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T07:16:09.851 INFO:teuthology.orchestra.run.smithi037.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T07:16:09.851 INFO:teuthology.orchestra.run.smithi037.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T07:16:09.851 INFO:teuthology.orchestra.run.smithi037.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T07:16:09.851 INFO:teuthology.orchestra.run.smithi037.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T07:16:09.851 INFO:teuthology.orchestra.run.smithi037.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2024-09-16T07:16:09.851 INFO:teuthology.orchestra.run.smithi037.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2024-09-16T07:16:09.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2024-09-16T07:16:09.851 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-16T07:16:09.852 INFO:teuthology.orchestra.run.smithi037.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2024-09-16T07:16:09.853 INFO:teuthology.orchestra.run.smithi037.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2024-09-16T07:16:09.853 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2024-09-16T07:16:09.853 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet 2024-09-16T07:16:09.853 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages will be upgraded: 2024-09-16T07:16:09.853 INFO:teuthology.orchestra.run.smithi037.stdout: librados2 librbd1 2024-09-16T07:16:09.923 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T07:16:09.924 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1focal [3,451 kB] 2024-09-16T07:16:09.926 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T07:16:09.944 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T07:16:09.997 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T07:16:09.999 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T07:16:10.002 INFO:teuthology.orchestra.run.smithi037.stdout:2 upgraded, 110 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T07:16:10.002 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 199 MB of archives. 2024-09-16T07:16:10.002 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 977 MB of additional disk space will be used. 2024-09-16T07:16:10.002 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1focal [3,451 kB] 2024-09-16T07:16:10.006 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T07:16:10.007 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T07:16:10.008 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T07:16:10.009 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T07:16:10.009 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T07:16:10.012 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T07:16:10.012 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T07:16:10.014 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T07:16:10.031 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T07:16:10.032 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T07:16:10.032 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T07:16:10.033 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T07:16:10.034 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T07:16:10.034 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T07:16:10.036 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T07:16:10.036 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T07:16:10.037 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T07:16:10.046 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T07:16:10.049 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T07:16:10.056 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T07:16:10.061 INFO:teuthology.orchestra.run.smithi143.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T07:16:10.061 INFO:teuthology.orchestra.run.smithi143.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T07:16:10.062 INFO:teuthology.orchestra.run.smithi143.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T07:16:10.063 INFO:teuthology.orchestra.run.smithi143.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T07:16:10.064 INFO:teuthology.orchestra.run.smithi143.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T07:16:10.067 INFO:teuthology.orchestra.run.smithi143.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T07:16:10.067 INFO:teuthology.orchestra.run.smithi143.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T07:16:10.069 INFO:teuthology.orchestra.run.smithi143.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T07:16:10.083 INFO:teuthology.orchestra.run.smithi143.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-16T07:16:10.084 INFO:teuthology.orchestra.run.smithi143.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T07:16:10.086 INFO:teuthology.orchestra.run.smithi143.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T07:16:10.086 INFO:teuthology.orchestra.run.smithi143.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T07:16:10.087 INFO:teuthology.orchestra.run.smithi143.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T07:16:10.087 INFO:teuthology.orchestra.run.smithi143.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T07:16:10.093 INFO:teuthology.orchestra.run.smithi143.stdout:Get:41 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1focal [4,042 kB] 2024-09-16T07:16:10.100 INFO:teuthology.orchestra.run.smithi143.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T07:16:10.101 INFO:teuthology.orchestra.run.smithi143.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T07:16:10.101 INFO:teuthology.orchestra.run.smithi143.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T07:16:10.102 INFO:teuthology.orchestra.run.smithi143.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T07:16:10.118 INFO:teuthology.orchestra.run.smithi143.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T07:16:10.118 INFO:teuthology.orchestra.run.smithi143.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T07:16:10.120 INFO:teuthology.orchestra.run.smithi143.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T07:16:10.122 INFO:teuthology.orchestra.run.smithi143.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T07:16:10.124 INFO:teuthology.orchestra.run.smithi143.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T07:16:10.125 INFO:teuthology.orchestra.run.smithi143.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T07:16:10.135 INFO:teuthology.orchestra.run.smithi143.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T07:16:10.164 INFO:teuthology.orchestra.run.smithi143.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1focal [746 kB] 2024-09-16T07:16:10.176 INFO:teuthology.orchestra.run.smithi143.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1focal [387 kB] 2024-09-16T07:16:10.199 INFO:teuthology.orchestra.run.smithi143.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1focal [31.9 kB] 2024-09-16T07:16:10.199 INFO:teuthology.orchestra.run.smithi143.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1focal [200 kB] 2024-09-16T07:16:10.203 INFO:teuthology.orchestra.run.smithi143.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1focal [62.5 kB] 2024-09-16T07:16:10.205 INFO:teuthology.orchestra.run.smithi143.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1focal [368 kB] 2024-09-16T07:16:10.212 INFO:teuthology.orchestra.run.smithi143.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1focal [6,630 kB] 2024-09-16T07:16:10.213 INFO:teuthology.orchestra.run.smithi143.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T07:16:10.230 INFO:teuthology.orchestra.run.smithi143.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T07:16:10.232 INFO:teuthology.orchestra.run.smithi143.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T07:16:10.243 INFO:teuthology.orchestra.run.smithi143.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T07:16:10.243 INFO:teuthology.orchestra.run.smithi143.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T07:16:10.243 INFO:teuthology.orchestra.run.smithi143.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T07:16:10.304 INFO:teuthology.orchestra.run.smithi143.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T07:16:10.305 INFO:teuthology.orchestra.run.smithi143.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T07:16:10.305 INFO:teuthology.orchestra.run.smithi143.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T07:16:10.306 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1focal [4,042 kB] 2024-09-16T07:16:10.306 INFO:teuthology.orchestra.run.smithi143.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T07:16:10.308 INFO:teuthology.orchestra.run.smithi143.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T07:16:10.309 INFO:teuthology.orchestra.run.smithi143.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T07:16:10.310 INFO:teuthology.orchestra.run.smithi143.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T07:16:10.315 INFO:teuthology.orchestra.run.smithi143.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T07:16:10.321 INFO:teuthology.orchestra.run.smithi143.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T07:16:10.336 INFO:teuthology.orchestra.run.smithi143.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T07:16:10.337 INFO:teuthology.orchestra.run.smithi143.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T07:16:10.340 INFO:teuthology.orchestra.run.smithi143.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T07:16:10.344 INFO:teuthology.orchestra.run.smithi143.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T07:16:10.350 INFO:teuthology.orchestra.run.smithi143.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1focal [113 kB] 2024-09-16T07:16:10.352 INFO:teuthology.orchestra.run.smithi143.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1focal [478 kB] 2024-09-16T07:16:10.359 INFO:teuthology.orchestra.run.smithi143.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1focal [25.2 MB] 2024-09-16T07:16:10.398 INFO:teuthology.orchestra.run.smithi143.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T07:16:10.403 INFO:teuthology.orchestra.run.smithi143.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-16T07:16:10.406 INFO:teuthology.orchestra.run.smithi143.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T07:16:10.449 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T07:16:10.449 INFO:teuthology.orchestra.run.smithi143.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T07:16:10.450 INFO:teuthology.orchestra.run.smithi143.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T07:16:10.450 INFO:teuthology.orchestra.run.smithi143.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T07:16:10.450 INFO:teuthology.orchestra.run.smithi143.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T07:16:10.465 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T07:16:10.465 INFO:teuthology.orchestra.run.smithi143.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T07:16:10.465 INFO:teuthology.orchestra.run.smithi143.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T07:16:10.465 INFO:teuthology.orchestra.run.smithi143.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T07:16:10.466 INFO:teuthology.orchestra.run.smithi143.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T07:16:10.466 INFO:teuthology.orchestra.run.smithi143.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T07:16:10.525 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1focal [746 kB] 2024-09-16T07:16:10.546 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T07:16:10.587 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1focal [387 kB] 2024-09-16T07:16:10.594 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1focal [31.9 kB] 2024-09-16T07:16:10.595 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1focal [200 kB] 2024-09-16T07:16:10.597 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1focal [62.5 kB] 2024-09-16T07:16:10.598 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1focal [368 kB] 2024-09-16T07:16:10.605 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1focal [6,630 kB] 2024-09-16T07:16:10.788 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T07:16:10.797 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T07:16:10.825 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T07:16:10.833 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T07:16:10.834 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T07:16:10.838 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T07:16:10.839 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T07:16:10.851 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T07:16:10.853 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T07:16:10.874 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T07:16:10.940 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T07:16:10.940 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T07:16:10.942 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T07:16:10.944 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T07:16:10.945 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T07:16:10.946 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T07:16:10.948 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T07:16:10.950 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T07:16:10.952 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T07:16:10.964 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1focal [113 kB] 2024-09-16T07:16:10.967 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1focal [478 kB] 2024-09-16T07:16:10.972 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1focal [25.2 MB] 2024-09-16T07:16:11.033 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T07:16:11.033 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T07:16:11.042 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T07:16:11.042 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T07:16:11.044 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T07:16:11.045 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T07:16:11.047 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T07:16:11.049 INFO:teuthology.orchestra.run.smithi037.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T07:16:11.091 INFO:teuthology.orchestra.run.smithi143.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1focal [9,611 kB] 2024-09-16T07:16:11.114 INFO:teuthology.orchestra.run.smithi037.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T07:16:11.119 INFO:teuthology.orchestra.run.smithi037.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T07:16:11.195 INFO:teuthology.orchestra.run.smithi037.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-16T07:16:11.198 INFO:teuthology.orchestra.run.smithi037.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T07:16:11.201 INFO:teuthology.orchestra.run.smithi037.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T07:16:11.203 INFO:teuthology.orchestra.run.smithi037.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T07:16:11.204 INFO:teuthology.orchestra.run.smithi037.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T07:16:11.205 INFO:teuthology.orchestra.run.smithi037.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T07:16:11.207 INFO:teuthology.orchestra.run.smithi037.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T07:16:11.208 INFO:teuthology.orchestra.run.smithi037.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T07:16:11.277 INFO:teuthology.orchestra.run.smithi037.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T07:16:11.277 INFO:teuthology.orchestra.run.smithi037.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T07:16:11.291 INFO:teuthology.orchestra.run.smithi037.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T07:16:11.292 INFO:teuthology.orchestra.run.smithi037.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T07:16:11.295 INFO:teuthology.orchestra.run.smithi037.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T07:16:11.298 INFO:teuthology.orchestra.run.smithi037.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T07:16:11.328 INFO:teuthology.orchestra.run.smithi143.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1focal [238 kB] 2024-09-16T07:16:11.329 INFO:teuthology.orchestra.run.smithi143.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1focal [148 kB] 2024-09-16T07:16:11.331 INFO:teuthology.orchestra.run.smithi143.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1focal [1,574 kB] 2024-09-16T07:16:11.359 INFO:teuthology.orchestra.run.smithi037.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T07:16:11.360 INFO:teuthology.orchestra.run.smithi037.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T07:16:11.361 INFO:teuthology.orchestra.run.smithi037.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T07:16:11.367 INFO:teuthology.orchestra.run.smithi143.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1focal [6,640 kB] 2024-09-16T07:16:11.575 INFO:teuthology.orchestra.run.smithi143.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1focal [24.5 MB] 2024-09-16T07:16:11.692 INFO:teuthology.orchestra.run.smithi037.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T07:16:11.718 INFO:teuthology.orchestra.run.smithi037.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T07:16:11.721 INFO:teuthology.orchestra.run.smithi037.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T07:16:11.791 INFO:teuthology.orchestra.run.smithi037.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T07:16:11.791 INFO:teuthology.orchestra.run.smithi037.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T07:16:11.792 INFO:teuthology.orchestra.run.smithi037.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T07:16:11.792 INFO:teuthology.orchestra.run.smithi037.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T07:16:11.793 INFO:teuthology.orchestra.run.smithi037.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T07:16:11.793 INFO:teuthology.orchestra.run.smithi037.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T07:16:11.794 INFO:teuthology.orchestra.run.smithi037.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T07:16:11.856 INFO:teuthology.orchestra.run.smithi037.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T07:16:11.902 INFO:teuthology.orchestra.run.smithi037.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T07:16:11.903 INFO:teuthology.orchestra.run.smithi037.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1focal [9,611 kB] 2024-09-16T07:16:11.938 INFO:teuthology.orchestra.run.smithi037.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T07:16:11.943 INFO:teuthology.orchestra.run.smithi037.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T07:16:11.947 INFO:teuthology.orchestra.run.smithi037.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T07:16:11.948 INFO:teuthology.orchestra.run.smithi037.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T07:16:11.949 INFO:teuthology.orchestra.run.smithi037.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T07:16:11.952 INFO:teuthology.orchestra.run.smithi037.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T07:16:11.954 INFO:teuthology.orchestra.run.smithi037.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T07:16:12.027 INFO:teuthology.orchestra.run.smithi037.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T07:16:12.033 INFO:teuthology.orchestra.run.smithi037.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-16T07:16:12.036 INFO:teuthology.orchestra.run.smithi037.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T07:16:12.198 INFO:teuthology.orchestra.run.smithi037.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T07:16:12.199 INFO:teuthology.orchestra.run.smithi037.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T07:16:12.264 INFO:teuthology.orchestra.run.smithi037.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T07:16:12.265 INFO:teuthology.orchestra.run.smithi037.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1focal [238 kB] 2024-09-16T07:16:12.266 INFO:teuthology.orchestra.run.smithi037.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T07:16:12.268 INFO:teuthology.orchestra.run.smithi037.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1focal [148 kB] 2024-09-16T07:16:12.271 INFO:teuthology.orchestra.run.smithi037.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1focal [1,574 kB] 2024-09-16T07:16:12.286 INFO:teuthology.orchestra.run.smithi037.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T07:16:12.287 INFO:teuthology.orchestra.run.smithi037.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T07:16:12.287 INFO:teuthology.orchestra.run.smithi037.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1focal [6,640 kB] 2024-09-16T07:16:12.288 INFO:teuthology.orchestra.run.smithi037.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T07:16:12.288 INFO:teuthology.orchestra.run.smithi037.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T07:16:12.289 INFO:teuthology.orchestra.run.smithi037.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T07:16:12.384 INFO:teuthology.orchestra.run.smithi143.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1focal [13.7 kB] 2024-09-16T07:16:12.426 INFO:teuthology.orchestra.run.smithi143.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1focal [856 kB] 2024-09-16T07:16:12.441 INFO:teuthology.orchestra.run.smithi143.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1focal [2,295 kB] 2024-09-16T07:16:12.448 INFO:teuthology.orchestra.run.smithi037.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1focal [24.5 MB] 2024-09-16T07:16:12.457 INFO:teuthology.orchestra.run.smithi143.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1focal [239 kB] 2024-09-16T07:16:12.459 INFO:teuthology.orchestra.run.smithi143.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1focal [143 kB] 2024-09-16T07:16:12.460 INFO:teuthology.orchestra.run.smithi143.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1focal [7,015 kB] 2024-09-16T07:16:12.658 INFO:teuthology.orchestra.run.smithi143.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1focal [7,882 kB] 2024-09-16T07:16:12.889 INFO:teuthology.orchestra.run.smithi143.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1focal [13.9 kB] 2024-09-16T07:16:12.890 INFO:teuthology.orchestra.run.smithi143.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1focal [46.9 MB] 2024-09-16T07:16:13.417 INFO:teuthology.orchestra.run.smithi037.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1focal [13.7 kB] 2024-09-16T07:16:13.418 INFO:teuthology.orchestra.run.smithi037.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1focal [856 kB] 2024-09-16T07:16:13.432 INFO:teuthology.orchestra.run.smithi037.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1focal [2,295 kB] 2024-09-16T07:16:13.463 INFO:teuthology.orchestra.run.smithi037.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1focal [239 kB] 2024-09-16T07:16:13.481 INFO:teuthology.orchestra.run.smithi037.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1focal [143 kB] 2024-09-16T07:16:13.484 INFO:teuthology.orchestra.run.smithi037.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1focal [7,015 kB] 2024-09-16T07:16:13.738 INFO:teuthology.orchestra.run.smithi037.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1focal [7,882 kB] 2024-09-16T07:16:14.049 INFO:teuthology.orchestra.run.smithi037.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1focal [13.9 kB] 2024-09-16T07:16:14.050 INFO:teuthology.orchestra.run.smithi037.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1focal [46.9 MB] 2024-09-16T07:16:14.489 INFO:teuthology.orchestra.run.smithi143.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1focal [120 kB] 2024-09-16T07:16:14.491 INFO:teuthology.orchestra.run.smithi143.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1focal [37.4 kB] 2024-09-16T07:16:14.492 INFO:teuthology.orchestra.run.smithi143.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1focal [13.5 MB] 2024-09-16T07:16:14.806 INFO:teuthology.orchestra.run.smithi143.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1focal [107 kB] 2024-09-16T07:16:15.388 INFO:teuthology.orchestra.run.smithi037.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1focal [120 kB] 2024-09-16T07:16:15.390 INFO:teuthology.orchestra.run.smithi037.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1focal [37.4 kB] 2024-09-16T07:16:15.391 INFO:teuthology.orchestra.run.smithi037.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1focal [13.5 MB] 2024-09-16T07:16:15.455 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 199 MB in 5s (39.6 MB/s) 2024-09-16T07:16:15.566 INFO:teuthology.orchestra.run.smithi037.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1focal [107 kB] 2024-09-16T07:16:15.650 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T07:16:15.707 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T07:16:15.712 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T07:16:15.815 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T07:16:16.203 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T07:16:16.221 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T07:16:16.236 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 199 MB in 6s (34.9 MB/s) 2024-09-16T07:16:16.261 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T07:16:16.412 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T07:16:16.466 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T07:16:16.470 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T07:16:16.550 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T07:16:16.656 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T07:16:16.672 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T07:16:16.715 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T07:16:16.937 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T07:16:16.957 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T07:16:16.995 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T07:16:17.159 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T07:16:17.177 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T07:16:17.251 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:16:17.315 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T07:16:17.334 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T07:16:17.373 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T07:16:17.776 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T07:16:17.780 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T07:16:17.795 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T07:16:17.797 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T07:16:17.848 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:16:17.854 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:16:18.241 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T07:16:18.261 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T07:16:18.299 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:16:18.346 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T07:16:18.362 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T07:16:18.421 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:16:18.719 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T07:16:18.737 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T07:16:18.778 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T07:16:18.782 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T07:16:18.801 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T07:16:18.840 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:16:19.165 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../007-librbd1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:19.177 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T07:16:19.193 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T07:16:19.235 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T07:16:19.325 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T07:16:19.612 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../007-librbd1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:19.738 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T07:16:20.026 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:20.186 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T07:16:20.409 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:20.517 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T07:16:20.948 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T07:16:20.966 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:21.032 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:21.238 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T07:16:21.254 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../009-libcephfs2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:21.296 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:21.368 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rados. 2024-09-16T07:16:21.386 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../010-python3-rados_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:21.417 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:21.623 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rados. 2024-09-16T07:16:21.639 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../010-python3-rados_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:21.681 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:21.796 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T07:16:21.813 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:21.887 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:22.034 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T07:16:22.052 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:22.109 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:22.165 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T07:16:22.181 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:22.223 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:22.337 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T07:16:22.356 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../012-python3-cephfs_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:22.387 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:22.576 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T07:16:22.591 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:22.626 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:22.697 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T07:16:22.714 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:22.756 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:22.946 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T07:16:22.965 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T07:16:23.038 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T07:16:23.092 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T07:16:23.110 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T07:16:23.142 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T07:16:23.286 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T07:16:23.302 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:23.349 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:23.395 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T07:16:23.411 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../015-python3-rbd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:23.461 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:23.735 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T07:16:23.751 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T07:16:23.793 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T07:16:23.831 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T07:16:23.850 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T07:16:23.889 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T07:16:24.221 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T07:16:24.238 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T07:16:24.242 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T07:16:24.261 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T07:16:24.280 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T07:16:24.325 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T07:16:24.582 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T07:16:24.600 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T07:16:24.603 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T07:16:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T07:16:24.632 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T07:16:24.661 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T07:16:24.997 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librgw2. 2024-09-16T07:16:25.013 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:25.044 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librgw2. 2024-09-16T07:16:25.062 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:25.102 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:25.950 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T07:16:25.966 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:26.008 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:26.030 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T07:16:26.046 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:26.088 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:26.344 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liboath0. 2024-09-16T07:16:26.363 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T07:16:26.402 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T07:16:26.691 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liboath0. 2024-09-16T07:16:26.709 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T07:16:26.741 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T07:16:26.938 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T07:16:26.956 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:26.996 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:27.102 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T07:16:27.119 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:27.169 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:27.467 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-common. 2024-09-16T07:16:27.484 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:27.525 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:27.689 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-common. 2024-09-16T07:16:27.706 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:27.747 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:29.988 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-base. 2024-09-16T07:16:30.006 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:30.065 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:30.212 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-base. 2024-09-16T07:16:30.230 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:30.280 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:31.318 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T07:16:31.334 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T07:16:31.376 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T07:16:31.633 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T07:16:31.650 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T07:16:31.687 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T07:16:31.691 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T07:16:31.707 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T07:16:31.770 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T07:16:32.061 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T07:16:32.081 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T07:16:32.082 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T07:16:32.100 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T07:16:32.120 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T07:16:32.140 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T07:16:32.434 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T07:16:32.448 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T07:16:32.450 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T07:16:32.468 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T07:16:32.506 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T07:16:32.509 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T07:16:32.784 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T07:16:32.799 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T07:16:32.834 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T07:16:32.854 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-mako. 2024-09-16T07:16:32.870 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T07:16:32.912 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T07:16:33.212 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-mako. 2024-09-16T07:16:33.229 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T07:16:33.231 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T07:16:33.251 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T07:16:33.270 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T07:16:33.281 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T07:16:33.559 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T07:16:33.579 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T07:16:33.617 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T07:16:33.623 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T07:16:33.644 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T07:16:33.682 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T07:16:33.903 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webob. 2024-09-16T07:16:33.923 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T07:16:33.978 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T07:16:34.001 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T07:16:34.020 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T07:16:34.084 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T07:16:34.306 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T07:16:34.322 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T07:16:34.364 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T07:16:34.379 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webob. 2024-09-16T07:16:34.395 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T07:16:34.429 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T07:16:34.650 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T07:16:34.667 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T07:16:34.717 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T07:16:34.749 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T07:16:34.766 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T07:16:34.807 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T07:16:35.020 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T07:16:35.037 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T07:16:35.073 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T07:16:35.143 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T07:16:35.162 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T07:16:35.226 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T07:16:35.365 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T07:16:35.381 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T07:16:35.448 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T07:16:35.613 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T07:16:35.630 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T07:16:35.675 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T07:16:35.726 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-paste. 2024-09-16T07:16:35.742 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T07:16:35.784 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T07:16:36.075 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T07:16:36.093 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T07:16:36.112 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T07:16:36.129 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T07:16:36.133 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T07:16:36.187 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T07:16:36.465 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T07:16:36.469 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-paste. 2024-09-16T07:16:36.481 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T07:16:36.487 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T07:16:36.523 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T07:16:36.528 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T07:16:36.811 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T07:16:36.831 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T07:16:36.890 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T07:16:36.890 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T07:16:36.906 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T07:16:36.973 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T07:16:37.209 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T07:16:37.227 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T07:16:37.268 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T07:16:37.318 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T07:16:37.334 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T07:16:37.376 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T07:16:37.623 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T07:16:37.640 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T07:16:37.690 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T07:16:37.704 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T07:16:37.720 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T07:16:37.786 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T07:16:37.976 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T07:16:37.992 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:38.034 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:38.124 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T07:16:38.142 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T07:16:38.182 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T07:16:38.456 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T07:16:38.475 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:38.514 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:38.545 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T07:16:38.565 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T07:16:38.612 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T07:16:38.808 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T07:16:38.827 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T07:16:38.858 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T07:16:38.990 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T07:16:39.007 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:39.048 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:39.186 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T07:16:39.204 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T07:16:39.244 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T07:16:39.446 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T07:16:39.465 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:39.537 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:39.606 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T07:16:39.626 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T07:16:39.664 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T07:16:39.900 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T07:16:39.918 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../045-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T07:16:39.959 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T07:16:40.046 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T07:16:40.062 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:40.096 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:40.281 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T07:16:40.301 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../046-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T07:16:40.339 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T07:16:40.603 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T07:16:40.623 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:40.666 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:40.768 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T07:16:40.784 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../047-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T07:16:40.826 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T07:16:41.162 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T07:16:41.181 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../048-ceph-mgr_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:41.212 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:41.627 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T07:16:41.642 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:41.684 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:41.690 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T07:16:41.707 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../049-ceph-mon_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:41.749 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:43.061 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T07:16:43.077 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../050-ceph-osd_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:43.110 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:44.029 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph. 2024-09-16T07:16:44.048 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../051-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:44.087 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:44.390 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T07:16:44.408 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:44.473 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:44.868 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T07:16:44.887 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:44.926 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:45.382 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph. 2024-09-16T07:16:45.398 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../051-ceph_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:45.440 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:45.471 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package cephadm. 2024-09-16T07:16:45.491 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../054-cephadm_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:45.529 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:45.817 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T07:16:45.834 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../052-ceph-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:45.849 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T07:16:45.865 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-16T07:16:45.876 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:45.907 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-16T07:16:46.252 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T07:16:46.271 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T07:16:46.309 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T07:16:46.321 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T07:16:46.338 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../053-ceph-mds_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:46.397 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:46.654 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T07:16:46.672 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:46.721 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:46.967 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package cephadm. 2024-09-16T07:16:46.983 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../054-cephadm_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:16:47.025 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:47.065 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T07:16:47.082 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T07:16:47.124 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T07:16:47.353 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T07:16:47.372 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../055-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-16T07:16:47.411 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-16T07:16:47.468 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-attr. 2024-09-16T07:16:47.488 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-16T07:16:47.526 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T07:16:47.873 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T07:16:47.879 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T07:16:47.890 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T07:16:47.897 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T07:16:47.931 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T07:16:47.946 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T07:16:48.276 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T07:16:48.296 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:48.298 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-py. 2024-09-16T07:16:48.315 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T07:16:48.334 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:48.357 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T07:16:48.660 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T07:16:48.679 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T07:16:48.718 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T07:16:48.729 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T07:16:48.746 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T07:16:48.787 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T07:16:49.046 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T07:16:49.062 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-16T07:16:49.104 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T07:16:49.132 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-attr. 2024-09-16T07:16:49.150 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2024-09-16T07:16:49.191 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T07:16:49.406 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T07:16:49.423 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T07:16:49.457 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T07:16:49.577 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T07:16:49.596 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T07:16:49.635 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T07:16:49.743 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T07:16:49.761 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T07:16:49.793 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T07:16:49.980 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-py. 2024-09-16T07:16:50.000 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T07:16:50.038 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T07:16:50.147 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T07:16:50.164 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T07:16:50.205 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T07:16:50.441 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T07:16:50.458 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T07:16:50.466 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T07:16:50.484 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T07:16:50.499 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T07:16:50.524 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T07:16:50.793 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T07:16:50.809 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T07:16:50.829 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T07:16:50.848 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2024-09-16T07:16:50.851 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T07:16:50.887 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T07:16:51.212 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-routes. 2024-09-16T07:16:51.223 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T07:16:51.229 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-16T07:16:51.240 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T07:16:51.270 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T07:16:51.298 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T07:16:51.598 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T07:16:51.615 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:51.634 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T07:16:51.650 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T07:16:51.656 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:51.692 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T07:16:52.071 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T07:16:52.089 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T07:16:52.154 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T07:16:52.441 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T07:16:52.460 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T07:16:52.499 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T07:16:52.801 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T07:16:52.817 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T07:16:52.860 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T07:16:53.011 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T07:16:53.031 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T07:16:53.069 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T07:16:53.204 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-routes. 2024-09-16T07:16:53.222 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2024-09-16T07:16:53.296 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T07:16:53.355 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T07:16:53.373 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T07:16:53.431 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T07:16:53.640 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T07:16:53.660 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:53.699 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:53.741 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T07:16:53.760 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T07:16:53.799 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T07:16:55.033 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T07:16:55.052 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T07:16:55.091 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T07:16:55.154 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T07:16:55.173 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T07:16:55.212 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T07:16:55.540 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T07:16:55.556 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T07:16:55.598 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T07:16:55.619 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T07:16:55.637 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T07:16:55.711 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T07:16:55.951 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T07:16:55.969 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T07:16:56.009 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T07:16:56.081 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T07:16:56.100 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T07:16:56.139 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T07:16:56.856 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T07:16:56.874 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:56.914 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:16:57.427 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T07:16:57.445 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T07:16:57.477 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T07:16:58.076 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T07:16:58.096 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T07:16:58.134 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T07:16:58.440 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T07:16:58.456 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T07:16:58.518 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T07:16:58.804 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T07:16:58.824 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T07:16:58.862 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T07:16:58.902 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T07:16:58.918 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T07:16:58.960 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T07:16:59.190 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T07:16:59.210 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T07:16:59.248 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T07:16:59.666 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T07:16:59.682 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-16T07:16:59.724 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T07:16:59.736 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T07:16:59.754 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:16:59.794 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:00.036 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T07:17:00.053 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T07:17:00.094 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T07:17:00.448 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T07:17:00.465 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T07:17:00.506 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T07:17:00.809 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T07:17:00.826 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T07:17:00.867 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T07:17:00.989 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T07:17:01.006 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T07:17:01.047 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T07:17:01.238 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T07:17:01.255 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T07:17:01.313 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T07:17:01.384 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T07:17:01.400 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T07:17:01.468 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T07:17:01.608 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T07:17:01.628 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T07:17:01.689 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T07:17:01.850 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T07:17:01.869 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T07:17:01.908 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T07:17:02.014 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T07:17:02.030 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T07:17:02.072 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T07:17:02.364 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T07:17:02.382 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2024-09-16T07:17:02.421 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T07:17:02.635 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T07:17:02.654 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:17:02.694 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:02.776 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T07:17:02.792 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T07:17:02.834 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T07:17:03.029 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T07:17:03.046 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T07:17:03.087 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T07:17:03.214 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T07:17:03.231 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T07:17:03.273 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T07:17:03.459 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T07:17:03.476 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T07:17:03.517 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:17:03.601 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T07:17:03.618 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T07:17:03.659 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T07:17:03.787 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package jq. 2024-09-16T07:17:03.806 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T07:17:03.845 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:17:03.967 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T07:17:03.983 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T07:17:04.017 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T07:17:04.106 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package socat. 2024-09-16T07:17:04.122 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T07:17:04.164 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T07:17:04.372 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T07:17:04.389 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T07:17:04.429 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T07:17:04.451 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T07:17:04.469 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T07:17:04.508 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T07:17:04.768 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T07:17:04.784 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T07:17:04.816 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T07:17:04.896 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-test. 2024-09-16T07:17:04.913 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../094-ceph-test_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:17:04.954 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:05.398 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T07:17:05.416 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:17:05.456 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:05.835 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T07:17:05.852 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T07:17:05.893 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T07:17:06.263 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T07:17:06.279 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T07:17:06.321 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:17:06.632 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package jq. 2024-09-16T07:17:06.650 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T07:17:06.715 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:17:07.002 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package socat. 2024-09-16T07:17:07.019 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T07:17:07.060 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T07:17:07.405 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T07:17:07.421 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T07:17:07.463 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T07:17:07.851 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-test. 2024-09-16T07:17:07.870 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../094-ceph-test_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:17:07.909 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:09.144 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T07:17:09.161 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:17:09.201 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:09.497 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T07:17:09.514 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T07:17:09.555 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:17:10.101 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T07:17:10.117 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:17:10.159 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:10.563 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T07:17:10.580 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T07:17:10.640 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T07:17:10.962 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T07:17:10.980 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-16T07:17:11.019 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-16T07:17:11.397 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T07:17:11.418 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T07:17:11.455 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:17:12.080 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T07:17:12.096 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../095-ceph-volume_18.2.4-845-ge4e227b3-1focal_all.deb ... 2024-09-16T07:17:12.139 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:12.462 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T07:17:12.481 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T07:17:12.545 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:17:13.112 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T07:17:13.113 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T07:17:13.132 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:17:13.134 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T07:17:13.170 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T07:17:13.174 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:13.523 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T07:17:13.540 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T07:17:13.581 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T07:17:13.586 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T07:17:13.603 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T07:17:13.661 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T07:17:13.867 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T07:17:13.886 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T07:17:13.977 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T07:17:13.986 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T07:17:14.005 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-16T07:17:14.036 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-16T07:17:14.304 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T07:17:14.322 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T07:17:14.362 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T07:17:14.423 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T07:17:14.439 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T07:17:14.480 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:17:14.840 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T07:17:14.858 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T07:17:14.898 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T07:17:15.227 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T07:17:15.245 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T07:17:15.285 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T07:17:15.607 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T07:17:15.628 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T07:17:15.657 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T07:17:15.951 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T07:17:15.972 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T07:17:16.018 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T07:17:16.131 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T07:17:16.149 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T07:17:16.189 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T07:17:16.321 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T07:17:16.338 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T07:17:16.379 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T07:17:16.591 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T07:17:16.613 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T07:17:16.650 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T07:17:16.977 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T07:17:16.995 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T07:17:17.035 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T07:17:17.083 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package radosgw. 2024-09-16T07:17:17.100 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../110-radosgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:17:17.140 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:17.397 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T07:17:17.417 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T07:17:17.480 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T07:17:18.019 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T07:17:18.038 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T07:17:18.079 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T07:17:18.413 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T07:17:18.433 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T07:17:18.471 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T07:17:18.560 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T07:17:18.581 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:17:18.618 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:18.824 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T07:17:18.845 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T07:17:18.907 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T07:17:18.925 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T07:17:19.243 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T07:17:19.251 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T07:17:19.262 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T07:17:19.301 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T07:17:19.377 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T07:17:19.478 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T07:17:19.604 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T07:17:19.620 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T07:17:19.662 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T07:17:19.679 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:20.222 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user cephadm....done 2024-09-16T07:17:20.387 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:17:20.459 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package radosgw. 2024-09-16T07:17:20.478 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../110-radosgw_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:17:20.512 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T07:17:20.517 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:20.864 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T07:17:21.157 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:17:21.266 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T07:17:21.392 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T07:17:21.752 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T07:17:21.945 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T07:17:21.963 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../111-rbd-fuse_18.2.4-845-ge4e227b3-1focal_amd64.deb ... 2024-09-16T07:17:22.003 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:22.154 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T07:17:22.335 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T07:17:22.464 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T07:17:22.666 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T07:17:22.757 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T07:17:22.792 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T07:17:22.938 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T07:17:23.025 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T07:17:23.129 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:23.326 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:23.618 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T07:17:23.721 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user cephadm....done 2024-09-16T07:17:23.769 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T07:17:23.901 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:17:24.027 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T07:17:24.062 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T07:17:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T07:17:24.421 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T07:17:24.506 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T07:17:24.722 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T07:17:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T07:17:24.882 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T07:17:24.999 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T07:17:25.183 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T07:17:25.325 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T07:17:25.484 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T07:17:25.727 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T07:17:25.835 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T07:17:26.062 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T07:17:26.170 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T07:17:26.421 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T07:17:26.504 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T07:17:26.722 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T07:17:26.805 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T07:17:27.023 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:27.115 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T07:17:27.240 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T07:17:27.324 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T07:17:27.350 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T07:17:27.450 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T07:17:27.659 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T07:17:27.784 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T07:17:28.086 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T07:17:28.143 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T07:17:28.269 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T07:17:28.471 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T07:17:28.662 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T07:17:28.831 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T07:17:28.982 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:29.422 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T07:17:29.708 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:17:29.714 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T07:17:29.842 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T07:17:30.057 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T07:17:30.202 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T07:17:30.328 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T07:17:30.349 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T07:17:30.471 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T07:17:30.709 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T07:17:30.756 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T07:17:31.052 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T07:17:31.065 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:17:31.208 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T07:17:31.334 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T07:17:31.361 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T07:17:31.487 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T07:17:31.646 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T07:17:31.669 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T07:17:31.947 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T07:17:31.953 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T07:17:32.238 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T07:17:32.339 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T07:17:32.357 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T07:17:32.632 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T07:17:32.733 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T07:17:32.974 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T07:17:33.135 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T07:17:33.269 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:33.284 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T07:17:33.671 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:17:33.682 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T07:17:33.830 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T07:17:33.945 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T07:17:34.063 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T07:17:34.181 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T07:17:34.340 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T07:17:34.414 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T07:17:34.474 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T07:17:34.773 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T07:17:34.834 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T07:17:34.925 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T07:17:35.051 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T07:17:35.169 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:17:35.294 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T07:17:35.385 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T07:17:35.445 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T07:17:35.662 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T07:17:35.788 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T07:17:36.089 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T07:17:36.390 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T07:17:36.508 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T07:17:36.842 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T07:17:37.151 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T07:17:37.158 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:17:37.284 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T07:17:37.477 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T07:17:37.661 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T07:17:37.848 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T07:17:37.962 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T07:17:38.130 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T07:17:38.247 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:17:38.281 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T07:17:38.373 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T07:17:38.607 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T07:17:38.716 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T07:17:38.983 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T07:17:39.051 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-16T07:17:39.135 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T07:17:39.260 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T07:17:39.436 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T07:17:39.611 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T07:17:39.737 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T07:17:39.954 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T07:17:40.801 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T07:17:40.952 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T07:17:41.053 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:17:41.154 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T07:17:41.455 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T07:17:41.475 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:17:41.601 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T07:17:41.614 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T07:17:41.924 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T07:17:42.028 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T07:17:42.225 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T07:17:42.337 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T07:17:42.568 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T07:17:42.647 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T07:17:42.772 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T07:17:42.912 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:43.021 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-16T07:17:43.167 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T07:17:43.172 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:43.298 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:43.493 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:43.509 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-16T07:17:43.603 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T07:17:43.871 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:43.910 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T07:17:43.980 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T07:17:44.290 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:44.316 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T07:17:44.416 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:44.542 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:44.812 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T07:17:44.812 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T07:17:45.254 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:45.349 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T07:17:45.380 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:45.500 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T07:17:45.506 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T07:17:45.626 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T07:17:45.793 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T07:17:46.111 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T07:17:46.237 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T07:17:46.434 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:46.560 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:46.563 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T07:17:46.670 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:46.779 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:46.914 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:46.923 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T07:17:47.232 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T07:17:47.458 INFO:teuthology.orchestra.run.smithi037.stdout:Adding group ceph....done 2024-09-16T07:17:47.593 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:47.718 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-16T07:17:47.852 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user ceph....done 2024-09-16T07:17:47.886 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:47.969 INFO:teuthology.orchestra.run.smithi037.stdout:Setting system user ceph properties....done 2024-09-16T07:17:47.991 INFO:teuthology.orchestra.run.smithi037.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T07:17:48.012 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:48.146 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T07:17:48.248 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:48.399 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T07:17:48.482 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T07:17:48.717 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:48.843 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T07:17:48.949 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:49.100 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:49.218 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:49.346 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:49.495 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:49.585 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T07:17:49.585 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T07:17:49.723 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T07:17:49.723 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T07:17:50.035 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:50.165 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:50.291 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:50.451 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T07:17:50.458 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T07:17:50.904 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:51.145 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T07:17:51.145 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T07:17:51.379 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:51.530 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:51.580 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:51.656 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:51.803 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:51.811 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T07:17:51.811 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T07:17:52.091 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:52.248 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:52.737 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T07:17:52.738 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T07:17:52.752 INFO:teuthology.orchestra.run.smithi143.stdout:Adding group ceph....done 2024-09-16T07:17:53.184 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:53.186 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user ceph....done 2024-09-16T07:17:53.309 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:53.379 INFO:teuthology.orchestra.run.smithi143.stdout:Setting system user ceph properties....done 2024-09-16T07:17:53.399 INFO:teuthology.orchestra.run.smithi143.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T07:17:53.469 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:53.552 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T07:17:53.690 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T07:17:53.691 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T07:17:53.878 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T07:17:54.203 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:54.341 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:54.355 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:54.480 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:54.500 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:54.623 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:54.982 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T07:17:54.982 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T07:17:55.035 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T07:17:55.186 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T07:17:55.428 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:55.480 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T07:17:55.832 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T07:17:56.271 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:56.496 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T07:17:56.496 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T07:17:56.947 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:57.199 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T07:17:57.199 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T07:17:57.648 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:58.154 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T07:17:58.155 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T07:17:58.366 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T07:17:58.600 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:58.751 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:58.902 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T07:17:58.987 DEBUG:teuthology.parallel:result is None 2024-09-16T07:17:59.161 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T07:17:59.161 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T07:17:59.620 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:59.754 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:17:59.905 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:18:00.047 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1focal) ... 2024-09-16T07:18:00.700 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T07:18:00.886 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T07:18:01.171 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T07:18:04.139 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T07:18:04.789 INFO:teuthology.orchestra.run.smithi143.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T07:18:04.794 DEBUG:teuthology.parallel:result is None 2024-09-16T07:18:04.795 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T07:18:04.943 DEBUG:teuthology.orchestra.run.smithi037:> dpkg-query -W -f '${Version}' ceph 2024-09-16T07:18:04.967 INFO:teuthology.orchestra.run.smithi037.stdout:18.2.4-845-ge4e227b3-1focal 2024-09-16T07:18:04.968 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1focal 2024-09-16T07:18:04.968 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1focal is installed. 2024-09-16T07:18:04.971 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T07:18:05.121 DEBUG:teuthology.orchestra.run.smithi143:> dpkg-query -W -f '${Version}' ceph 2024-09-16T07:18:05.144 INFO:teuthology.orchestra.run.smithi143.stdout:18.2.4-845-ge4e227b3-1focal 2024-09-16T07:18:05.144 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1focal 2024-09-16T07:18:05.144 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1focal is installed. 2024-09-16T07:18:05.146 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T07:18:05.146 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:05.146 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T07:18:05.163 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:05.163 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T07:18:05.179 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T07:18:05.179 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:05.179 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T07:18:05.195 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T07:18:05.255 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:05.255 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T07:18:05.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T07:18:05.285 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T07:18:05.285 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:05.285 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T07:18:05.301 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T07:18:05.313 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:05.313 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T07:18:05.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T07:18:05.385 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T07:18:05.385 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:05.385 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T07:18:05.401 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T07:18:05.414 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:05.414 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T07:18:05.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T07:18:05.439 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T07:18:05.525 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T07:18:05.525 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /var/log/ceph 2024-09-16T07:18:05.528 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /var/log/ceph 2024-09-16T07:18:05.543 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T07:18:05.544 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T07:18:05.584 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T07:18:05.600 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T07:18:05.600 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T07:18:05.602 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T07:18:05.667 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T07:18:05.667 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T07:18:05.667 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621', 'branch': 'reef', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} thrashers/sync-many workloads/pool-create-delete}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906621', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 931, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 1390, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8', 'sleep_before_teardown': 0, 'subset': '5319/100000', 'suite': 'rados', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6', 'targets': {'smithi037.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi143.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1, 'thrash_many': True}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_22:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T07:18:05.668 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T07:18:05.670 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T07:18:05.676 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T07:18:05.720 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T07:18:05.736 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:05.736 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T07:18:05.741 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T07:18:05.741 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 2024-09-16T07:18:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T07:18:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:18:05.747 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 6h/6d Inode: 358 Links: 1 2024-09-16T07:18:05.748 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:18:05.748 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-16 07:14:28.794920081 +0000 2024-09-16T07:18:05.748 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-16 07:14:28.598924590 +0000 2024-09-16T07:18:05.748 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-16 07:14:28.598924590 +0000 2024-09-16T07:18:05.748 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-09-16T07:18:05.749 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T07:18:05.803 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-16T07:18:05.803 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-16T07:18:05.803 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000421351 s, 1.2 MB/s 2024-09-16T07:18:05.804 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T07:18:05.812 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 2024-09-16T07:18:05.818 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T07:18:05.818 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:18:05.818 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 6h/6d Inode: 368 Links: 1 2024-09-16T07:18:05.818 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:18:05.818 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-16 07:14:29.234909958 +0000 2024-09-16T07:18:05.819 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-16 07:14:29.026914744 +0000 2024-09-16T07:18:05.819 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-16 07:14:29.026914744 +0000 2024-09-16T07:18:05.819 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-09-16T07:18:05.819 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T07:18:05.832 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-16T07:18:05.832 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-16T07:18:05.833 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000400635 s, 1.3 MB/s 2024-09-16T07:18:05.834 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T07:18:05.890 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 2024-09-16T07:18:05.897 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T07:18:05.897 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:18:05.897 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-09-16T07:18:05.897 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:18:05.897 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-16 07:14:29.686899560 +0000 2024-09-16T07:18:05.897 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-16 07:14:29.458904804 +0000 2024-09-16T07:18:05.897 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-16 07:14:29.458904804 +0000 2024-09-16T07:18:05.897 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-09-16T07:18:05.898 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T07:18:05.909 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-16T07:18:05.909 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-16T07:18:05.909 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000420635 s, 1.2 MB/s 2024-09-16T07:18:05.910 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T07:18:05.917 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 2024-09-16T07:18:05.965 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T07:18:05.965 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:18:05.965 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 6h/6d Inode: 379 Links: 1 2024-09-16T07:18:05.965 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:18:05.965 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-16 07:14:30.130889344 +0000 2024-09-16T07:18:05.965 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-16 07:14:29.930893946 +0000 2024-09-16T07:18:05.965 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-16 07:14:29.930893946 +0000 2024-09-16T07:18:05.965 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-09-16T07:18:05.965 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T07:18:05.976 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-16T07:18:05.976 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-16T07:18:05.976 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000375858 s, 1.4 MB/s 2024-09-16T07:18:05.977 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T07:18:05.985 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T07:18:05.985 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:05.985 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T07:18:05.992 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T07:18:05.992 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2024-09-16T07:18:05.999 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T07:18:05.999 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:18:05.999 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 361 Links: 1 2024-09-16T07:18:05.999 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:18:05.999 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-16 07:14:30.499940327 +0000 2024-09-16T07:18:05.999 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-16 07:14:30.495940418 +0000 2024-09-16T07:18:05.999 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-16 07:14:30.495940418 +0000 2024-09-16T07:18:06.000 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-09-16T07:18:06.000 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T07:18:06.057 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-16T07:18:06.057 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-16T07:18:06.057 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000429224 s, 1.2 MB/s 2024-09-16T07:18:06.058 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T07:18:06.066 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2024-09-16T07:18:06.072 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T07:18:06.072 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:18:06.073 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 365 Links: 1 2024-09-16T07:18:06.073 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:18:06.073 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-16 07:14:31.123925944 +0000 2024-09-16T07:18:06.073 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-16 07:14:30.911930831 +0000 2024-09-16T07:18:06.073 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-16 07:14:30.911930831 +0000 2024-09-16T07:18:06.073 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-09-16T07:18:06.073 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T07:18:06.129 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-16T07:18:06.129 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-16T07:18:06.129 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000400698 s, 1.3 MB/s 2024-09-16T07:18:06.130 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T07:18:06.138 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2024-09-16T07:18:06.144 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T07:18:06.144 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:18:06.144 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 372 Links: 1 2024-09-16T07:18:06.144 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:18:06.144 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-16 07:14:31.551916079 +0000 2024-09-16T07:18:06.144 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-16 07:14:31.339920965 +0000 2024-09-16T07:18:06.144 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-16 07:14:31.339920965 +0000 2024-09-16T07:18:06.145 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-09-16T07:18:06.145 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T07:18:06.201 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-16T07:18:06.201 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-16T07:18:06.201 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000421194 s, 1.2 MB/s 2024-09-16T07:18:06.202 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T07:18:06.210 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2024-09-16T07:18:06.216 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T07:18:06.216 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T07:18:06.216 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 379 Links: 1 2024-09-16T07:18:06.216 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T07:18:06.217 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-16 07:14:31.775910915 +0000 2024-09-16T07:18:06.217 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-16 07:14:31.771911008 +0000 2024-09-16T07:18:06.217 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-16 07:14:31.771911008 +0000 2024-09-16T07:18:06.217 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2024-09-16T07:18:06.217 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T07:18:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-16T07:18:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-16T07:18:06.273 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000360225 s, 1.4 MB/s 2024-09-16T07:18:06.274 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T07:18:06.282 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T07:18:06.283 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T07:18:06.283 INFO:tasks.ceph:Generating config... 2024-09-16T07:18:06.284 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T07:18:06.284 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T07:18:06.284 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T07:18:06.284 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-16T07:18:06.284 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T07:18:06.284 INFO:tasks.ceph:[global] ms type = async 2024-09-16T07:18:06.284 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T07:18:06.285 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T07:18:06.286 INFO:tasks.ceph:[mon] paxos min = 10 2024-09-16T07:18:06.286 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T07:18:06.286 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-09-16T07:18:06.286 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T07:18:06.286 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T07:18:06.286 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T07:18:06.286 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T07:18:06.286 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T07:18:06.286 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T07:18:06.286 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T07:18:06.286 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T07:18:06.286 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T07:18:06.287 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T07:18:06.287 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T07:18:06.287 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T07:18:06.287 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T07:18:06.287 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T07:18:06.288 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T07:18:06.288 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T07:18:06.288 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T07:18:06.288 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T07:18:06.288 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-16T07:18:06.288 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T07:18:06.288 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T07:18:06.288 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T07:18:06.288 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T07:18:06.288 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T07:18:06.331 INFO:teuthology.orchestra.run.smithi037.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T07:18:06.345 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T07:18:06.387 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T07:18:06.398 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.37'), ('mon.c', '[v2:172.21.15.37:3301,v1:172.21.15.37:6790]'), ('mon.b', '172.21.15.143')] 2024-09-16T07:18:06.398 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.37,[v2:172.21.15.37:3301,v1:172.21.15.37:6790],172.21.15.143', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T07:18:06.399 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:06.399 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T07:18:06.449 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.37 --addv c '[v2:172.21.15.37:3301,v1:172.21.15.37:6790]' --add b 172.21.15.143 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:06.494 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:06.494 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: generated fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:06.494 INFO:teuthology.orchestra.run.smithi037.stdout:setting min_mon_release = pacific 2024-09-16T07:18:06.494 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 0 2024-09-16T07:18:06.494 INFO:teuthology.orchestra.run.smithi037.stdout:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:06.494 INFO:teuthology.orchestra.run.smithi037.stdout:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:06.494 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:06.494 INFO:teuthology.orchestra.run.smithi037.stdout:min_mon_release 16 (pacific) 2024-09-16T07:18:06.495 INFO:teuthology.orchestra.run.smithi037.stdout:election_strategy: 3 2024-09-16T07:18:06.495 INFO:teuthology.orchestra.run.smithi037.stdout:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:06.495 INFO:teuthology.orchestra.run.smithi037.stdout:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:06.495 INFO:teuthology.orchestra.run.smithi037.stdout:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:06.495 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T07:18:06.496 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T07:18:06.500 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2b3c1781-6e4a-42f3-a6cb-6f1008258949... 2024-09-16T07:18:06.501 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T07:18:06.504 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout: chdir = "" 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout: auth supported = cephx 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout: filestore xattr use omap = true 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout: mon clock drift allowed = 1.000 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout: osd crush chooseleaf type = 0 2024-09-16T07:18:06.537 INFO:teuthology.orchestra.run.smithi037.stdout: auth debug = true 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on old message = true 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on bug = true 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: mon pg warn max object skew = 0 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default size = 2 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.538 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow primary affinity = true 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow pg remap = true 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on legacy crush tunables = false 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on crush straw calc version zero = false 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on no sortbitwise = false 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on osd down out interval zero = false 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on too few osds = false 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: mon_allow_pool_size_one = true 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: osd default data pool replay window = 5 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.539 INFO:teuthology.orchestra.run.smithi037.stdout: mon allow pool delete = true 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: mon cluster log file level = debug 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: debug asserts on shutdown = true 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: mon health detail to clog = false 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: mon host = "172.21.15.37,[v2:172.21.15.37:3301,v1:172.21.15.37:6790],172.21.15.143" 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: mon client directed command retry = 5 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: mon election default strategy = 3 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject socket failures = 5000 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: ms type = async 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: fsid = 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout:[osd] 2024-09-16T07:18:06.540 INFO:teuthology.orchestra.run.smithi037.stdout: osd journal size = 100 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub load threshold = 5.0 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub max interval = 600 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock profile = high_recovery_ops 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: osd recover clone overlap = true 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: osd recovery max chunk = 1048576 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug shutdown = true 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug op order = true 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify stray on activate = true 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: osd open classes on start = true 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug pg log writeout = true 2024-09-16T07:18:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: osd deep scrub update digest min age = 30 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: osd map max advance = 10 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: journal zero on create = true 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: filestore ondisk finisher threads = 3 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: filestore apply finisher threads = 3 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: bdev debug aio = true 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug misdirected ops = true 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: bdev async discard = True 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: bdev enable discard = True 2024-09-16T07:18:06.542 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore block size = 96636764160 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore fsck on mount = True 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore zero block detection = True 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluefs = 20 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluestore = 20 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: debug osd = 20 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: debug rocksdb = 10 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd full ratio = 0.9 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd nearfull ratio = 0.8 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify cached snaps = True 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify missing on start = True 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: osd failsafe full ratio = 0.95 2024-09-16T07:18:06.543 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock override recovery settings = True 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: osd memory target = 1610612736 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: osd objectstore = bluestore 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue = debug_random 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue cut off = debug_random 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr] 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: debug mgr = 20 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-09-16T07:18:06.544 INFO:teuthology.orchestra.run.smithi037.stdout: mgr/telemetry/nag = false 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout:[mon] 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: debug paxos = 20 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: mon data avail warn = 5 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: mon mgr mkfs grace = 240 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd reporter subtree level = osd 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd prime pg temp = true 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-09-16T07:18:06.545 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: # 1m isn't quite enough 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: mon_down_mkfs_grace = 2m 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_filestore_osds = false 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: mon min osdmap epochs = 25 2024-09-16T07:18:06.546 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune interval = 2 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune min = 15 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune txsize = 2 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub inject missing keys = 0.05 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub interval = 300 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: paxos min = 10 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: paxos service trim min = 5 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: paxos trim min = 10 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout:[client] 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: rgw cache enabled = true 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable ops log = true 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable usage log = true 2024-09-16T07:18:06.547 INFO:teuthology.orchestra.run.smithi037.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T07:18:06.548 INFO:teuthology.orchestra.run.smithi037.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T07:18:06.548 INFO:teuthology.orchestra.run.smithi037.stdout: debug monc = 20 2024-09-16T07:18:06.548 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-09-16T07:18:06.548 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.a] 2024-09-16T07:18:06.548 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.c] 2024-09-16T07:18:06.548 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.b] 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2024-09-16T07:18:06.549 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.550 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: mon_allow_pool_size_one = true 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2024-09-16T07:18:06.551 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = "172.21.15.37,[v2:172.21.15.37:3301,v1:172.21.15.37:6790],172.21.15.143" 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: mon client directed command retry = 5 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: mon election default strategy = 3 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject socket failures = 5000 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: ms type = async 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2024-09-16T07:18:06.552 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: osd mclock profile = high_recovery_ops 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2024-09-16T07:18:06.553 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2024-09-16T07:18:06.554 INFO:teuthology.orchestra.run.smithi143.stdout: bdev async discard = True 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: bdev enable discard = True 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore zero block detection = True 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 20 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 20 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 10 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify cached snaps = True 2024-09-16T07:18:06.555 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify missing on start = True 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: osd mclock override recovery settings = True 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: osd memory target = 1610612736 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: osd op queue = debug_random 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: osd op queue cut off = debug_random 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2024-09-16T07:18:06.556 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2024-09-16T07:18:06.557 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: # 1m isn't quite enough 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: mon_down_mkfs_grace = 2m 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_filestore_osds = false 2024-09-16T07:18:06.558 INFO:teuthology.orchestra.run.smithi143.stdout: mon min osdmap epochs = 25 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune interval = 2 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune min = 15 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune txsize = 2 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout: mon scrub inject missing keys = 0.05 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout: mon scrub interval = 300 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout: paxos min = 10 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout: paxos service trim min = 5 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout: paxos trim min = 10 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-16T07:18:06.559 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2024-09-16T07:18:06.560 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2024-09-16T07:18:06.560 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2024-09-16T07:18:06.560 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2024-09-16T07:18:06.560 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T07:18:06.561 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T07:18:06.561 INFO:teuthology.orchestra.run.smithi143.stdout: debug monc = 20 2024-09-16T07:18:06.561 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-16T07:18:06.561 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2024-09-16T07:18:06.561 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.c] 2024-09-16T07:18:06.561 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.b] 2024-09-16T07:18:06.562 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T07:18:06.562 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T07:18:06.658 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T07:18:06.659 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:06.659 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T07:18:06.666 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:06.666 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T07:18:06.672 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T07:18:06.672 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:06.672 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T07:18:06.673 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T07:18:06.739 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:06.739 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:06.746 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T07:18:06.746 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:06.746 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T07:18:06.746 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T07:18:06.772 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:06.772 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:06.779 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T07:18:06.779 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T07:18:06.779 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T07:18:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T07:18:06.842 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T07:18:06.842 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T07:18:06.843 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T07:18:06.882 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T07:18:06.895 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T07:18:06.895 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T07:18:06.895 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T07:18:06.911 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T07:18:06.911 INFO:tasks.ceph:role: osd.0 2024-09-16T07:18:06.912 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T07:18:06.912 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T07:18:07.438 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T07:18:07.439 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T07:18:07.439 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T07:18:07.439 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 2024-09-16T07:18:07.439 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T07:18:07.439 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-16T07:18:07.439 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T07:18:07.439 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T07:18:07.439 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T07:18:07.439 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T07:18:07.440 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-16T07:18:07.440 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T07:18:07.545 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T07:18:07.557 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T07:18:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:18:07.558 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T07:18:07.618 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T07:18:07.618 INFO:tasks.ceph:role: osd.1 2024-09-16T07:18:07.618 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T07:18:07.619 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T07:18:08.117 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T07:18:08.117 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T07:18:08.117 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T07:18:08.117 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 2024-09-16T07:18:08.117 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T07:18:08.117 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-16T07:18:08.117 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T07:18:08.117 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T07:18:08.117 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T07:18:08.117 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T07:18:08.119 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-16T07:18:08.119 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T07:18:08.154 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T07:18:08.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:18:08.165 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T07:18:08.165 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T07:18:08.220 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T07:18:08.220 INFO:tasks.ceph:role: osd.2 2024-09-16T07:18:08.220 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T07:18:08.220 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T07:18:08.697 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T07:18:08.698 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T07:18:08.698 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T07:18:08.698 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 2024-09-16T07:18:08.698 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T07:18:08.698 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-16T07:18:08.699 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T07:18:08.699 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T07:18:08.699 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T07:18:08.699 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T07:18:08.701 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-16T07:18:08.701 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T07:18:08.744 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T07:18:08.756 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T07:18:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:18:08.757 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:08.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:08.889+0000 7f3096ae7880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T07:18:08.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:08.889+0000 7f3096ae7880 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T07:18:08.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:08.889+0000 7f3096ae7880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T07:18:08.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:08.889+0000 7f3096ae7880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T07:18:08.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:08.889+0000 7f3096ae7880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T07:18:08.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:08.893+0000 7f3096ae7880 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T07:18:11.910 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T07:18:11.928 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:11.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:11.973+0000 7ff1325f6880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T07:18:11.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:11.973+0000 7ff1325f6880 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T07:18:11.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:11.973+0000 7ff1325f6880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T07:18:11.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:11.973+0000 7ff1325f6880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T07:18:11.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:11.973+0000 7ff1325f6880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T07:18:11.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:11.973+0000 7ff1325f6880 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T07:18:14.949 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T07:18:14.966 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:15.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:15.057+0000 7ff38d485880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T07:18:15.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:15.057+0000 7ff38d485880 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T07:18:15.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:15.057+0000 7ff38d485880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T07:18:15.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:15.057+0000 7ff38d485880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T07:18:15.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:15.057+0000 7ff38d485880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T07:18:15.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:15.057+0000 7ff38d485880 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T07:18:18.073 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T07:18:18.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T07:18:18.106 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T07:18:18.106 INFO:tasks.ceph:role: osd.3 2024-09-16T07:18:18.106 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T07:18:18.106 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T07:18:18.616 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T07:18:18.616 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T07:18:18.616 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T07:18:18.617 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2024-09-16T07:18:18.617 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T07:18:18.617 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-09-16T07:18:18.617 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T07:18:18.617 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T07:18:18.617 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T07:18:18.617 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T07:18:18.618 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-09-16T07:18:18.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T07:18:18.717 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T07:18:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:18:18.728 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T07:18:18.729 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T07:18:18.738 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T07:18:18.739 INFO:tasks.ceph:role: osd.4 2024-09-16T07:18:18.739 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T07:18:18.739 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T07:18:19.264 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T07:18:19.264 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T07:18:19.264 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T07:18:19.264 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2024-09-16T07:18:19.264 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T07:18:19.264 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-09-16T07:18:19.264 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T07:18:19.265 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T07:18:19.265 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T07:18:19.265 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T07:18:19.266 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-09-16T07:18:19.266 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T07:18:19.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T07:18:19.338 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T07:18:19.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:18:19.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T07:18:19.351 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T07:18:19.351 INFO:tasks.ceph:role: osd.5 2024-09-16T07:18:19.351 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T07:18:19.351 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T07:18:19.821 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T07:18:19.821 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T07:18:19.821 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T07:18:19.821 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 2024-09-16T07:18:19.821 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T07:18:19.821 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-09-16T07:18:19.821 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T07:18:19.821 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T07:18:19.821 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T07:18:19.821 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T07:18:19.822 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-09-16T07:18:19.822 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T07:18:19.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T07:18:19.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:18:19.885 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T07:18:19.886 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:19.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:19.966+0000 7f4380b05880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T07:18:19.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:19.966+0000 7f4380b05880 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T07:18:19.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:19.966+0000 7f4380b05880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T07:18:19.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:19.966+0000 7f4380b05880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T07:18:19.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:19.966+0000 7f4380b05880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-16T07:18:19.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:19.966+0000 7f4380b05880 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T07:18:23.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T07:18:23.032 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:23.119 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:23.118+0000 7ff4e264f880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T07:18:23.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:23.118+0000 7ff4e264f880 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T07:18:23.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:23.118+0000 7ff4e264f880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T07:18:23.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:23.118+0000 7ff4e264f880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T07:18:23.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:23.118+0000 7ff4e264f880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-16T07:18:23.121 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:23.118+0000 7ff4e264f880 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T07:18:26.134 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T07:18:26.150 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:26.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:26.190+0000 7fc30c0a6880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T07:18:26.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:26.190+0000 7fc30c0a6880 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T07:18:26.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:26.190+0000 7fc30c0a6880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T07:18:26.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:26.190+0000 7fc30c0a6880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T07:18:26.192 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:26.190+0000 7fc30c0a6880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-16T07:18:26.193 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:26.190+0000 7fc30c0a6880 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T07:18:29.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T07:18:29.246 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T07:18:29.247 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:29.247 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T07:18:29.264 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:29.264 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T07:18:29.280 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:29.280 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T07:18:29.338 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:29.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T07:18:29.354 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:29.354 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T07:18:29.363 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:29.364 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T07:18:29.422 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:29.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T07:18:29.432 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:29.433 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T07:18:29.437 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T07:18:29.437 DEBUG:teuthology.orchestra.run.smithi037:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T07:18:29.440 DEBUG:teuthology.orchestra.run.smithi143:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T07:18:29.449 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.0] 2024-09-16T07:18:29.449 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCw2+dmDiItNRAARSZhqH7ufsNDDHJyQP0fPg== 2024-09-16T07:18:29.449 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.1] 2024-09-16T07:18:29.449 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCz2+dm0gAqOhAAm9/kdhxfphMu92XPoEzHiQ== 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.2] 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQC32+dmxWOuAxAA1KD6Djp9CMyFkRW2bykhwQ== 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.x] 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCu2+dm5Dc5MRAAmsv6EjbHOflBpkTZiPxpng== 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.3] 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQC72+dme42vORAAhLw8laYwfUMdJGEV3UpPGw== 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.4] 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQC/2+dm2I8uBxAA8Hq34WWH8nBPsD6GjrXccw== 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.5] 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDC2+dm2ZB7CxAAubr8x9FaePZCmOx/bjqlCA== 2024-09-16T07:18:29.450 INFO:teuthology.orchestra.run.smithi143.stdout:[client.0] 2024-09-16T07:18:29.451 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCu2+dmf/GeNBAAkeTJUzqTZhdqLLegSorUpw== 2024-09-16T07:18:29.452 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.0] 2024-09-16T07:18:29.452 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCw2+dmDiItNRAARSZhqH7ufsNDDHJyQP0fPg== 2024-09-16T07:18:29.452 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.1] 2024-09-16T07:18:29.452 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCz2+dm0gAqOhAAm9/kdhxfphMu92XPoEzHiQ== 2024-09-16T07:18:29.452 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.2] 2024-09-16T07:18:29.452 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQC32+dmxWOuAxAA1KD6Djp9CMyFkRW2bykhwQ== 2024-09-16T07:18:29.452 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.x] 2024-09-16T07:18:29.453 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCu2+dm5Dc5MRAAmsv6EjbHOflBpkTZiPxpng== 2024-09-16T07:18:29.453 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.3] 2024-09-16T07:18:29.453 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQC72+dme42vORAAhLw8laYwfUMdJGEV3UpPGw== 2024-09-16T07:18:29.453 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.4] 2024-09-16T07:18:29.453 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQC/2+dm2I8uBxAA8Hq34WWH8nBPsD6GjrXccw== 2024-09-16T07:18:29.453 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.5] 2024-09-16T07:18:29.453 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDC2+dm2ZB7CxAAubr8x9FaePZCmOx/bjqlCA== 2024-09-16T07:18:29.453 INFO:teuthology.orchestra.run.smithi037.stdout:[client.0] 2024-09-16T07:18:29.453 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCu2+dmf/GeNBAAkeTJUzqTZhdqLLegSorUpw== 2024-09-16T07:18:29.455 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.552 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.600 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.651 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T07:18:29.653 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T07:18:29.747 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.750 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.800 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.852 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T07:18:29.901 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T07:18:29.903 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T07:18:29.953 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T07:18:29.953 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T07:18:29.965 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T07:18:30.048 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T07:18:30.065 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T07:18:30.075 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T07:18:30.185 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T07:18:30.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T07:18:30.217 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T07:18:30.436 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T07:18:30.453 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:30.456 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T07:18:30.461 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T07:18:30.461 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T07:18:30.461 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T07:18:30.463 INFO:tasks.ceph.mon.a:Started 2024-09-16T07:18:30.464 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T07:18:30.464 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T07:18:30.466 INFO:tasks.ceph.mon.c:Started 2024-09-16T07:18:30.466 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T07:18:30.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T07:18:30.468 INFO:tasks.ceph.mon.b:Started 2024-09-16T07:18:30.468 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T07:18:30.469 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T07:18:30.469 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T07:18:30.471 INFO:tasks.ceph.mgr.x:Started 2024-09-16T07:18:30.471 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T07:18:30.471 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T07:18:30.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.593+0000 7ff86a944700 10 monclient: get_monmap_and_config 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.593+0000 7ff86a944700 10 monclient: build_initial_monmap 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.593+0000 7ff86a944700 10 monclient: monmap: 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:30.598692+0000 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:30.598692+0000 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:30.595 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:30.596 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:30.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.593+0000 7ff86a944700 1 Processor -- start 2024-09-16T07:18:30.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 1 -- start start 2024-09-16T07:18:30.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 10 monclient: init 2024-09-16T07:18:30.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:30.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T07:18:30.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff864071ad0 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:30.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff86409a0f0 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 10 monclient(hunting): picked mon.noname-b con 0x7ff86409a500 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff864071eb0 con 0x7ff86409a0f0 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8640a02c0 con 0x7ff864071ad0 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86409cc40 con 0x7ff86409a500 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff86a944700 10 monclient(hunting): authenticate will time out at 831.085285s 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff8637fe700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff86409a0f0 0x7ff8641371b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51804/0 (socket says 172.21.15.37:51804) 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff8637fe700 1 -- 172.21.15.37:0/359618170 learned_addr learned my addr 172.21.15.37:0/359618170 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:30.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff868ee1700 10 monclient(hunting): get_auth_request con 0x7ff86409a500 auth_method 0 2024-09-16T07:18:30.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff868ee1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:30.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff868ee1700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:30.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff868ee1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:30.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 msgr2=0x7ff86413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:18:30.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 msgr2=0x7ff86413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:30.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:30.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.597+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:18:30.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.801+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:30.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.801+0000 7ff868ee1700 10 monclient(hunting): get_auth_request con 0x7ff86409a500 auth_method 0 2024-09-16T07:18:30.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.801+0000 7ff868ee1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:30.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.801+0000 7ff868ee1700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:30.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.801+0000 7ff868ee1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:30.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.801+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 msgr2=0x7ff86413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:18:30.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.801+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 msgr2=0x7ff86413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:30.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.801+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:30.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:30.801+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:18:31.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:31.205+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:31.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:31.205+0000 7ff868ee1700 10 monclient(hunting): get_auth_request con 0x7ff86409a500 auth_method 0 2024-09-16T07:18:31.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:31.205+0000 7ff868ee1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:31.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:31.205+0000 7ff868ee1700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:31.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:31.205+0000 7ff868ee1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:31.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:31.205+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 msgr2=0x7ff86413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:18:31.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:31.205+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 msgr2=0x7ff86413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:31.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:31.205+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:31.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:31.205+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:18:32.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:32.009+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:32.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:32.009+0000 7ff868ee1700 10 monclient(hunting): get_auth_request con 0x7ff86409a500 auth_method 0 2024-09-16T07:18:32.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:32.009+0000 7ff868ee1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:32.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:32.009+0000 7ff868ee1700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:32.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:32.009+0000 7ff868ee1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:32.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:32.009+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 msgr2=0x7ff86413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:18:32.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:32.009+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 msgr2=0x7ff86413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:32.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:32.009+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:32.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:32.009+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T07:18:33.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 10 monclient(hunting): tick 2024-09-16T07:18:33.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T07:18:33.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 1 monclient(hunting): continuing hunt 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 msgr2=0x7ff86413a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff86409a500 0x7ff86413a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 1 -- 172.21.15.37:0/359618170 >> v1:172.21.15.143:6789/0 conn(0x7ff864071ad0 legacy=0x7ff8640999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 1 -- 172.21.15.37:0/359618170 >> v1:172.21.15.37:6789/0 conn(0x7ff86409a0f0 legacy=0x7ff8641371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 10 monclient: _add_conns ranks=[3,4] 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 10 monclient(hunting): picked mon.noname-a con 0x7ff844005980 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 10 monclient(hunting): picked mon.noname-c con 0x7ff8440084d0 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 1 -- 172.21.15.37:0/359618170 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86409cc40 con 0x7ff844005980 2024-09-16T07:18:33.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 1 -- 172.21.15.37:0/359618170 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84400b030 con 0x7ff8440084d0 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff861ffb700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff8637fe700 10 monclient(hunting): get_auth_request con 0x7ff8440084d0 auth_method 0 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff8637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff8637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff8637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff863fff700 10 monclient(hunting): get_auth_request con 0x7ff844005980 auth_method 0 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff863fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff863fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff863fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff8637fe700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 msgr2=0x7ff84400a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T07:18:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff8637fe700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 msgr2=0x7ff84400a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff863fff700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 msgr2=0x7ff844007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:18:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff863fff700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 msgr2=0x7ff844007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:18:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.597+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:18:33.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff863fff700 10 monclient(hunting): get_auth_request con 0x7ff844005980 auth_method 0 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff863fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff863fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff863fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff8637fe700 10 monclient(hunting): get_auth_request con 0x7ff8440084d0 auth_method 0 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff8637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff8637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff8637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff863fff700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 msgr2=0x7ff844007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff863fff700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 msgr2=0x7ff844007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff8637fe700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 msgr2=0x7ff84400a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:18:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff8637fe700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 msgr2=0x7ff84400a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:18:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:33.801+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:18:34.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff863fff700 10 monclient(hunting): get_auth_request con 0x7ff844005980 auth_method 0 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff863fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff863fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff863fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff863fff700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 msgr2=0x7ff844007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff863fff700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 msgr2=0x7ff844007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff8637fe700 10 monclient(hunting): get_auth_request con 0x7ff8440084d0 auth_method 0 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff8637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff8637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff8637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:34.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:18:34.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff8637fe700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 msgr2=0x7ff84400a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:18:34.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff8637fe700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 msgr2=0x7ff84400a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:34.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:34.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:34.205+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:18:35.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:35.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff863fff700 10 monclient(hunting): get_auth_request con 0x7ff844005980 auth_method 0 2024-09-16T07:18:35.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff863fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:35.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff863fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:35.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff863fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:35.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff863fff700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 msgr2=0x7ff844007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T07:18:35.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:35.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff863fff700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 msgr2=0x7ff844007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:35.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:35.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff8637fe700 10 monclient(hunting): get_auth_request con 0x7ff8440084d0 auth_method 0 2024-09-16T07:18:35.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff8637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:35.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff8637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:35.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff8637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:35.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T07:18:35.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff8637fe700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 msgr2=0x7ff84400a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:18:35.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff8637fe700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 msgr2=0x7ff84400a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:35.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:35.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:35.009+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T07:18:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 10 monclient(hunting): tick 2024-09-16T07:18:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 10 monclient(hunting): _check_auth_tickets 2024-09-16T07:18:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 1 monclient(hunting): continuing hunt 2024-09-16T07:18:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T07:18:36.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 msgr2=0x7ff84400a970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:18:36.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.143:3300/0 conn(0x7ff8440084d0 0x7ff84400a970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 msgr2=0x7ff844007e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:18:36.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff844005980 0x7ff844007e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T07:18:36.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 reap_dead start 2024-09-16T07:18:36.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff86409a0f0 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:36.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff844005df0 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:36.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff84400a080 0x7ff84400a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:36.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 10 monclient(hunting): picked mon.noname-a con 0x7ff84400a080 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:36.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 1 -- 172.21.15.37:0/359618170 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86409bf80 con 0x7ff86409a0f0 2024-09-16T07:18:36.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 1 -- 172.21.15.37:0/359618170 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84400aab0 con 0x7ff844005df0 2024-09-16T07:18:36.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:36.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 1 -- 172.21.15.37:0/359618170 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86409cc40 con 0x7ff84400a080 2024-09-16T07:18:36.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff861ffb700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:36.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff84400a080 0x7ff84400a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:36.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff868ee1700 10 monclient(hunting): get_auth_request con 0x7ff84400a080 auth_method 0 2024-09-16T07:18:36.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff868ee1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:36.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff868ee1700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:36.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff868ee1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:36.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff84400a080 msgr2=0x7ff84400a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:18:36.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff84400a080 msgr2=0x7ff84400a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:18:36.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff84400a080 0x7ff84400a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:18:36.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.597+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff84400a080 0x7ff84400a530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:18:36.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752990124 0 0) 0x7ff84400aab0 con 0x7ff844005df0 2024-09-16T07:18:36.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): _negotiate 2024-09-16T07:18:36.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:36.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:36.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): my global_id is 4102 2024-09-16T07:18:36.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8500035d0 con 0x7ff844005df0 2024-09-16T07:18:36.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879063206 0 0) 0x7ff86409bf80 con 0x7ff86409a0f0 2024-09-16T07:18:36.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): _negotiate 2024-09-16T07:18:36.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:36.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:36.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): my global_id is 4098 2024-09-16T07:18:36.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff84400aab0 con 0x7ff86409a0f0 2024-09-16T07:18:36.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3902598405 0 0) 0x7ff84400aab0 con 0x7ff86409a0f0 2024-09-16T07:18:36.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff84400a080 msgr2=0x7ff84400a530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:18:36.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 --2- 172.21.15.37:0/359618170 >> v2:172.21.15.37:3300/0 conn(0x7ff84400a080 0x7ff84400a530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 >> v1:172.21.15.143:6789/0 conn(0x7ff844005df0 legacy=0x7ff844008960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:36.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient: _finish_hunting 0 2024-09-16T07:18:36.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 monclient: found mon.noname-a-legacy 2024-09-16T07:18:36.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:36.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-09-16T07:18:36.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84400ad10 con 0x7ff86409a0f0 2024-09-16T07:18:36.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient: _finish_auth 0 2024-09-16T07:18:36.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:36.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 120013200 0 0) 0x7ff85c001c50 con 0x7ff86409a0f0 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient: dump: 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:36.675 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 >> v1:172.21.15.37:6789/0 conn(0x7ff86409a0f0 legacy=0x7ff84400d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff850003bb0 0x7ff850003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): picked mon.b con 0x7ff850003bb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850004870 0x7ff850004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): picked mon.c con 0x7ff850004870 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff850005350 0x7ff85000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): picked mon.a con 0x7ff850005350 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:36.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86409cc40 con 0x7ff850005350 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85000c200 con 0x7ff850003bb0 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85000c3c0 con 0x7ff850004870 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff862ffd700 10 monclient(hunting): _finish_auth 0 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff850005350 0x7ff85000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff850003bb0 0x7ff850003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850004870 0x7ff850004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff863fff700 10 monclient(hunting): get_auth_request con 0x7ff850003bb0 auth_method 0 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff863fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff863fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:36.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff863fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient(hunting): get_auth_request con 0x7ff850005350 auth_method 0 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff8637fe700 10 monclient(hunting): get_auth_request con 0x7ff850004870 auth_method 0 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff8637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff8637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff8637fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:36.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850004870 msgr2=0x7ff850004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850004870 0x7ff850004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff850003bb0 msgr2=0x7ff850003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff850003bb0 0x7ff850003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient: _finish_hunting 0 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 1 monclient: found mon.a 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84400aab0 con 0x7ff850005350 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient: _finish_auth 0 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 10 monclient: _check_auth_tickets 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff86a944700 5 monclient: authenticate success, global_id 4098 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff86a944700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff863fff700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff850003bb0 0x7ff850003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff8637fe700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850004870 0x7ff850004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 1 -- 172.21.15.37:0/359618170 reap_dead start 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.669+0000 7ff868ee1700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff850005350 0x7ff85000ba30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff86409b600 tx=0x7ff8500058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e494a8bc2579e60 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c004220 con 0x7ff850005350 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff862ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff862ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff862ffd700 10 monclient: dump: 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:36.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c00c040 con 0x7ff850005350 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff862ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff862ffd700 1 -- 172.21.15.37:0/359618170 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c004580 con 0x7ff850005350 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff862ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 4 set_mon_vals no callback set 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff862ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff862ffd700 10 monclient: dump: 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:36.682 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:36.683 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:36.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:36.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:36.683 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:36.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 10 monclient: shutdown 2024-09-16T07:18:36.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:36.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 1 -- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff850005350 msgr2=0x7ff85000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:36.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff850005350 0x7ff85000ba30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff86409b600 tx=0x7ff8500058a0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 1 -- 172.21.15.37:0/359618170 shutdown_connections 2024-09-16T07:18:36.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 1 --2- 172.21.15.37:0/359618170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff850005350 0x7ff85000ba30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 1 -- 172.21.15.37:0/359618170 >> 172.21.15.37:0/359618170 conn(0x7ff8640a04b0 msgr2=0x7ff8640a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 1 -- 172.21.15.37:0/359618170 shutdown_connections 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 1 -- 172.21.15.37:0/359618170 wait complete. 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.673+0000 7ff86a944700 10 monclient: build_initial_monmap 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient: monmap: 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:36.688 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:36.689 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:36.689 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:36.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 1 Processor -- start 2024-09-16T07:18:36.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 1 -- start start 2024-09-16T07:18:36.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient: init 2024-09-16T07:18:36.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:36.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:18:36.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850005350 0x7ff8641c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:36.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient(hunting): picked mon.noname-c con 0x7ff850005350 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:36.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8641c4050 0x7ff8641ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:36.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient(hunting): picked mon.noname-a con 0x7ff8641c4050 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:36.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8641c44d0 0x7ff8641ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:36.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient(hunting): picked mon.noname-b con 0x7ff8641c44d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:36.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:36.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8641ab110 con 0x7ff8641c4050 2024-09-16T07:18:36.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:36.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640708e0 con 0x7ff8641c44d0 2024-09-16T07:18:36.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:36.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff864072660 con 0x7ff850005350 2024-09-16T07:18:36.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:36.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient(hunting): authenticate will time out at 837.165681s 2024-09-16T07:18:36.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850005350 0x7ff8641c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:36.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8637fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8641c4050 0x7ff8641ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:36.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850005350 0x7ff8641c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:54962/0 (socket says 172.21.15.37:54962) 2024-09-16T07:18:36.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8637fe700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8641c4050 0x7ff8641ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47238/0 (socket says 172.21.15.37:47238) 2024-09-16T07:18:36.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 1 -- 172.21.15.37:0/831439200 learned_addr learned my addr 172.21.15.37:0/831439200 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:36.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 10 monclient(hunting): get_auth_request con 0x7ff850005350 auth_method 0 2024-09-16T07:18:36.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:36.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:36.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:36.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff868ee1700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8641c44d0 0x7ff8641ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:36.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8637fe700 10 monclient(hunting): get_auth_request con 0x7ff8641c4050 auth_method 0 2024-09-16T07:18:36.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:36.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8637fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:36.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:36.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff868ee1700 10 monclient(hunting): get_auth_request con 0x7ff8641c44d0 auth_method 0 2024-09-16T07:18:36.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff868ee1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:36.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff868ee1700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:36.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff868ee1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:36.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:36.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:36.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:36.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff868ee1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:36.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff868ee1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:36.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff868ee1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:36.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:36.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 10 monclient(hunting): handle_auth_done global_id 4100 payload 824 2024-09-16T07:18:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 1 -- 172.21.15.37:0/831439200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8641c44d0 msgr2=0x7ff8641ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8641c44d0 0x7ff8641ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 1 -- 172.21.15.37:0/831439200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8641c4050 msgr2=0x7ff8641ca000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:36.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8641c4050 0x7ff8641ca000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 10 monclient: _finish_hunting 0 2024-09-16T07:18:36.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 1 monclient: found mon.noname-c 2024-09-16T07:18:36.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:36.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:18:36.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 1 -- 172.21.15.37:0/831439200 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8641a8e80 con 0x7ff850005350 2024-09-16T07:18:36.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff868ee1700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8641c44d0 0x7ff8641ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:36.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8637fe700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8641c4050 0x7ff8641ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:36.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff863fff700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850005350 0x7ff8641c6ac0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff85c005670 tx=0x7ff85c005980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:36.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 1 -- 172.21.15.37:0/831439200 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c0159f0 con 0x7ff850005350 2024-09-16T07:18:36.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:36.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:36.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 10 monclient: dump: 2024-09-16T07:18:36.698 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:36.698 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:36.698 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:36.698 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 10 monclient: _finish_auth 0 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 1 -- 172.21.15.37:0/831439200 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85c015b50 con 0x7ff850005350 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 5 monclient: authenticate success, global_id 4100 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient: _renew_subs 2024-09-16T07:18:36.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 1 -- 172.21.15.37:0/831439200 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8640a3260 con 0x7ff850005350 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient: _renew_subs 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 1 -- 172.21.15.37:0/831439200 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c015ee0 con 0x7ff850005350 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff86a944700 1 -- 172.21.15.37:0/831439200 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8641a9670 con 0x7ff850005350 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff8627fc700 4 set_mon_vals no callback set 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 10 monclient: dump: 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:36.700 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.677+0000 7ff860ff9700 1 -- 172.21.15.37:0/831439200 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7ff85c017430 con 0x7ff850005350 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.681+0000 7ff860ff9700 1 -- 172.21.15.37:0/831439200 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff85c01f8d0 con 0x7ff850005350 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.681+0000 7ff86a944700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:36.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.681+0000 7ff86a944700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:36.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.681+0000 7ff86a944700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:36.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.681+0000 7ff86a944700 1 -- 172.21.15.37:0/831439200 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8641a04d0 con 0x7ff850005350 2024-09-16T07:18:36.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.685+0000 7ff860ff9700 1 -- 172.21.15.37:0/831439200 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7ff85c01fba0 con 0x7ff850005350 2024-09-16T07:18:36.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.685+0000 7ff860ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:36.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.685+0000 7ff860ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.757+0000 7ff86a944700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T07:18:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.757+0000 7ff86a944700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T07:18:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.757+0000 7ff86a944700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.757+0000 7ff86a944700 1 -- 172.21.15.37:0/831439200 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff8640999e0 con 0x7ff850005350 2024-09-16T07:18:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff860ff9700 1 -- 172.21.15.37:0/831439200 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff85c01fd40 con 0x7ff850005350 2024-09-16T07:18:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff860ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T07:18:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff860ff9700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T07:18:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 10 monclient: shutdown 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 1 -- 172.21.15.37:0/831439200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850005350 msgr2=0x7ff8641c6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850005350 0x7ff8641c6ac0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff85c005670 tx=0x7ff85c005980 comp rx=0 tx=0).stop 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 1 -- 172.21.15.37:0/831439200 shutdown_connections 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8641c44d0 0x7ff8641ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8641c4050 0x7ff8641ca000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 1 --2- 172.21.15.37:0/831439200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff850005350 0x7ff8641c6ac0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 1 -- 172.21.15.37:0/831439200 >> 172.21.15.37:0/831439200 conn(0x7ff8640a04b0 msgr2=0x7ff8640a1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 1 -- 172.21.15.37:0/831439200 shutdown_connections 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:36.765+0000 7ff86a944700 1 -- 172.21.15.37:0/831439200 wait complete. 2024-09-16T07:18:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:adjusted tunables profile to default 2024-09-16T07:18:36.778 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T07:18:36.779 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T07:18:36.779 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:36.779 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T07:18:36.793 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:36.793 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T07:18:36.848 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-16T07:18:36.848 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T07:18:36.859 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:36.859 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T07:18:36.876 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:36.876 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T07:18:36.938 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-16T07:18:36.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T07:18:36.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new fd764fc8-b5fb-40fe-a64f-ce7b8c1df525 0 2024-09-16T07:18:37.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.062+0000 7f0a7759e700 10 monclient: get_monmap_and_config 2024-09-16T07:18:37.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.062+0000 7f0a7759e700 10 monclient: build_initial_monmap 2024-09-16T07:18:37.066 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.062+0000 7f0a7759e700 10 monclient: monmap: 2024-09-16T07:18:37.066 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:37.067040+0000 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:37.067040+0000 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:37.067 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 Processor -- start 2024-09-16T07:18:37.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 -- start start 2024-09-16T07:18:37.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient: init 2024-09-16T07:18:37.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:37.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T07:18:37.070 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a780a2750 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0a780a3240 0x7f0a781382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient(hunting): picked mon.noname-c con 0x7f0a780a3240 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0a78138830 0x7f0a7813ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient(hunting): picked mon.noname-a con 0x7f0a78138830 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a78099be0 con 0x7f0a780a2750 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7809de40 con 0x7f0a78138830 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7809d530 con 0x7f0a780a3240 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient(hunting): authenticate will time out at 832.744958s 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0a780a3240 0x7f0a781382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.071 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7659c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f0a780a2750 0x7f0a780a2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:50964/0 (socket says 172.21.15.143:50964) 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7659c700 1 -- 172.21.15.143:0/2650613831 learned_addr learned my addr 172.21.15.143:0/2650613831 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 10 monclient(hunting): get_auth_request con 0x7f0a780a3240 auth_method 0 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7659c700 1 -- 172.21.15.143:0/2650613831 >> v1:172.21.15.143:6789/0 conn(0x7f0a780a2750 legacy=0x7f0a780a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a76d9d700 1 --2- 172.21.15.143:0/2650613831 >> v2:172.21.15.37:3300/0 conn(0x7f0a78138830 0x7f0a7813ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a76d9d700 10 monclient(hunting): get_auth_request con 0x7f0a78138830 auth_method 0 2024-09-16T07:18:37.072 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a76d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a76d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a76d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 1 -- 172.21.15.143:0/2650613831 >> v2:172.21.15.37:3300/0 conn(0x7f0a78138830 msgr2=0x7f0a7813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 1 --2- 172.21.15.143:0/2650613831 >> v2:172.21.15.37:3300/0 conn(0x7f0a78138830 0x7f0a7813ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 1 -- 172.21.15.143:0/2650613831 >> v1:172.21.15.143:6789/0 conn(0x7f0a780a2750 legacy=0x7f0a780a2b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 10 monclient: _finish_hunting 0 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 1 monclient: found mon.noname-c 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 1 -- 172.21.15.143:0/2650613831 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7809a020 con 0x7f0a780a3240 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a75d9b700 1 --2- 172.21.15.143:0/2650613831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a3240 0x7f0a781382f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0a780a3650 tx=0x7f0a6c009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e767054e1b82320b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 1 -- 172.21.15.143:0/2650613831 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c004230 con 0x7f0a780a3240 2024-09-16T07:18:37.073 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 10 monclient: dump: 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.074 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 10 monclient: _finish_auth 0 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 10 monclient: _check_auth_tickets 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 1 -- 172.21.15.143:0/2650613831 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c004390 con 0x7f0a780a3240 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 5 monclient: authenticate success, global_id 4111 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 1 -- 172.21.15.143:0/2650613831 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0036c0 con 0x7f0a780a3240 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 4 set_mon_vals no callback set 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7559a700 10 monclient: dump: 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.075 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient: shutdown 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 -- 172.21.15.143:0/2650613831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a3240 msgr2=0x7f0a781382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 --2- 172.21.15.143:0/2650613831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a3240 0x7f0a781382f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0a780a3650 tx=0x7f0a6c009b30 comp rx=0 tx=0).stop 2024-09-16T07:18:37.076 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 -- 172.21.15.143:0/2650613831 shutdown_connections 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 --2- 172.21.15.143:0/2650613831 >> v2:172.21.15.37:3300/0 conn(0x7f0a78138830 0x7f0a7813ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 --2- 172.21.15.143:0/2650613831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a3240 0x7f0a781382f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 -- 172.21.15.143:0/2650613831 >> 172.21.15.143:0/2650613831 conn(0x7f0a780a3ea0 msgr2=0x7f0a780a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 -- 172.21.15.143:0/2650613831 shutdown_connections 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 1 -- 172.21.15.143:0/2650613831 wait complete. 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient: build_initial_monmap 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.066+0000 7f0a7759e700 10 monclient: monmap: 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:37.077 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T07:18:37.078 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:37.078 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:37.078 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:37.078 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 Processor -- start 2024-09-16T07:18:37.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 -- start start 2024-09-16T07:18:37.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient: init 2024-09-16T07:18:37.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:37.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:18:37.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a2750 0x7f0a781c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.082 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient(hunting): picked mon.noname-b con 0x7f0a780a2750 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a780a3240 0x7f0a78137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient(hunting): picked mon.noname-c con 0x7f0a780a3240 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a78138830 0x7f0a781c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient(hunting): picked mon.noname-a con 0x7f0a78138830 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a781221f0 con 0x7f0a78138830 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7806fb80 con 0x7f0a780a2750 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a78121f10 con 0x7f0a780a3240 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient(hunting): authenticate will time out at 832.749118s 2024-09-16T07:18:37.083 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a2750 0x7f0a781c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a2750 0x7f0a781c29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45202/0 (socket says 172.21.15.143:45202) 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 1 -- 172.21.15.143:0/973425391 learned_addr learned my addr 172.21.15.143:0/973425391 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 10 monclient(hunting): get_auth_request con 0x7f0a780a2750 auth_method 0 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a75d9b700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a780a3240 0x7f0a78137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a76d9d700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a78138830 0x7f0a781c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-09-16T07:18:37.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 1 -- 172.21.15.143:0/973425391 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a780a3240 msgr2=0x7f0a78137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a780a3240 0x7f0a78137e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 1 -- 172.21.15.143:0/973425391 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a78138830 msgr2=0x7f0a781c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a78138830 0x7f0a781c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 10 monclient: _finish_hunting 0 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 1 monclient: found mon.noname-b 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 1 -- 172.21.15.143:0/973425391 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0094b0 con 0x7f0a780a2750 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a75d9b700 10 monclient: get_auth_request con 0x7f0a780a3240 auth_method 0 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a75d9b700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a780a3240 0x7f0a78137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a76d9d700 10 monclient: get_auth_request con 0x7f0a78138830 auth_method 0 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a76d9d700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a78138830 0x7f0a781c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T07:18:37.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7659c700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a2750 0x7f0a781c29e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800c550 tx=0x7f0a6800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 1 -- 172.21.15.143:0/973425391 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68005d90 con 0x7f0a780a2750 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 10 monclient: dump: 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.086 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 10 monclient: _finish_auth 0 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 5 monclient: authenticate success, global_id 4114 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 1 -- 172.21.15.143:0/973425391 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a680070d0 con 0x7f0a780a2750 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 1 -- 172.21.15.143:0/973425391 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68007480 con 0x7f0a780a2750 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient: _renew_subs 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 -- 172.21.15.143:0/973425391 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7809b710 con 0x7f0a780a2750 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.087 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a74d99700 4 set_mon_vals no callback set 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 10 monclient: dump: 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.088 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient: _renew_subs 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 -- 172.21.15.143:0/973425391 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a781227b0 con 0x7f0a780a2750 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 1 -- 172.21.15.143:0/973425391 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f0a6801c440 con 0x7f0a780a2750 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a66ffd700 1 -- 172.21.15.143:0/973425391 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0a68012810 con 0x7f0a780a2750 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.070+0000 7f0a7759e700 1 -- 172.21.15.143:0/973425391 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a780a2b30 con 0x7f0a780a2750 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.074+0000 7f0a66ffd700 1 -- 172.21.15.143:0/973425391 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f0a68017070 con 0x7f0a780a2750 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.074+0000 7f0a66ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.089 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.074+0000 7f0a66ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:37.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.150+0000 7f0a7759e700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fd764fc8-b5fb-40fe-a64f-ce7b8c1df525", "id": 0}] 2024-09-16T07:18:37.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.150+0000 7f0a7759e700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fd764fc8-b5fb-40fe-a64f-ce7b8c1df525", "id": 0}] 2024-09-16T07:18:37.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.150+0000 7f0a7759e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.150+0000 7f0a7759e700 1 -- 172.21.15.143:0/973425391 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fd764fc8-b5fb-40fe-a64f-ce7b8c1df525", "id": 0} v 0) v1 -- 0x7f0a78070d70 con 0x7f0a780a2750 2024-09-16T07:18:37.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.158+0000 7f0a66ffd700 1 -- 172.21.15.143:0/973425391 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fd764fc8-b5fb-40fe-a64f-ce7b8c1df525", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0a68012af0 con 0x7f0a780a2750 2024-09-16T07:18:37.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.158+0000 7f0a66ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fd764fc8-b5fb-40fe-a64f-ce7b8c1df525", "id": 0}] 2024-09-16T07:18:37.161 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.158+0000 7f0a66ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:37.161 INFO:teuthology.orchestra.run.smithi143.stdout:0 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 10 monclient: shutdown 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 1 -- 172.21.15.143:0/973425391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a2750 msgr2=0x7f0a781c29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a2750 0x7f0a781c29e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800c550 tx=0x7f0a6800c910 comp rx=0 tx=0).stop 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 1 -- 172.21.15.143:0/973425391 shutdown_connections 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a78138830 0x7f0a781c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a780a3240 0x7f0a78137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 1 --2- 172.21.15.143:0/973425391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a780a2750 0x7f0a781c29e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 1 -- 172.21.15.143:0/973425391 >> 172.21.15.143:0/973425391 conn(0x7f0a780a3ea0 msgr2=0x7f0a7806f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 1 -- 172.21.15.143:0/973425391 shutdown_connections 2024-09-16T07:18:37.163 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.162+0000 7f0a7759e700 1 -- 172.21.15.143:0/973425391 wait complete. 2024-09-16T07:18:37.179 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 52d577a5-08b7-452d-aa35-e1560215a09e 1 2024-09-16T07:18:37.286 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:37.282+0000 7f8b087a32c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T07:18:37.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient: get_monmap_and_config 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient: build_initial_monmap 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient: monmap: 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:37.287713+0000 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:37.287713+0000 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:37.287 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 1 Processor -- start 2024-09-16T07:18:37.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 1 -- start start 2024-09-16T07:18:37.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient: init 2024-09-16T07:18:37.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:37.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-16T07:18:37.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7ea8135810 0x7f7ea8135bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient(hunting): picked mon.noname-c con 0x7f7ea8135810 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:37.296 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ea809e0b0 0x7f7ea806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient(hunting): picked mon.noname-b con 0x7f7ea809e0b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ea80a0ab0 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea8136130 con 0x7f7ea80a0ab0 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea806d660 con 0x7f7ea8135810 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80a0f80 con 0x7f7ea809e0b0 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient(hunting): authenticate will time out at 832.965562s 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7ea8135810 0x7f7ea8135bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7ea8135810 0x7f7ea8135bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45210/0 (socket says 172.21.15.143:45210) 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 1 -- 172.21.15.143:0/3546416319 learned_addr learned my addr 172.21.15.143:0/3546416319 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T07:18:37.297 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 10 monclient(hunting): get_auth_request con 0x7f7ea8135810 auth_method 0 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7ead20d700 1 --2- 172.21.15.143:0/3546416319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ea809e0b0 0x7f7ea806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7ead20d700 10 monclient(hunting): get_auth_request con 0x7f7ea809e0b0 auth_method 0 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7ead20d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7ead20d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7ead20d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7ead20d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.298 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7ead20d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7ead20d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 1 -- 172.21.15.143:0/3546416319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ea809e0b0 msgr2=0x7f7ea806a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 1 --2- 172.21.15.143:0/3546416319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ea809e0b0 0x7f7ea806a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 1 -- 172.21.15.143:0/3546416319 >> v1:172.21.15.37:6789/0 conn(0x7f7ea80a0ab0 legacy=0x7f7ea806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 10 monclient: _finish_hunting 0 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 1 monclient: found mon.noname-c 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 1 -- 172.21.15.143:0/3546416319 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea809ea20 con 0x7f7ea8135810 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eada0e700 1 --2- 172.21.15.143:0/3546416319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea8135810 0x7f7ea8135bf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009910 tx=0x7f7e98009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30be7dbf916a0d27 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 1 -- 172.21.15.143:0/3546416319 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98011830 con 0x7f7ea8135810 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.299 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 10 monclient: dump: 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 10 monclient: _finish_auth 0 2024-09-16T07:18:37.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 10 monclient: _check_auth_tickets 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 1 -- 172.21.15.143:0/3546416319 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e98011990 con 0x7f7ea8135810 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 5 monclient: authenticate success, global_id 4117 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 1 -- 172.21.15.143:0/3546416319 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98011d40 con 0x7f7ea8135810 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 4 set_mon_vals no callback set 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaca0c700 10 monclient: dump: 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.303 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.303 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.303 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.303 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.303 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.303 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.303 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 10 monclient: shutdown 2024-09-16T07:18:37.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 1 -- 172.21.15.143:0/3546416319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea8135810 msgr2=0x7f7ea8135bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.286+0000 7f7eaea10700 1 --2- 172.21.15.143:0/3546416319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea8135810 0x7f7ea8135bf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009910 tx=0x7f7e98009c20 comp rx=0 tx=0).stop 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 -- 172.21.15.143:0/3546416319 shutdown_connections 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 --2- 172.21.15.143:0/3546416319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ea809e0b0 0x7f7ea806a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 --2- 172.21.15.143:0/3546416319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea8135810 0x7f7ea8135bf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 -- 172.21.15.143:0/3546416319 >> 172.21.15.143:0/3546416319 conn(0x7f7ea80a14c0 msgr2=0x7f7ea80a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 -- 172.21.15.143:0/3546416319 shutdown_connections 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 -- 172.21.15.143:0/3546416319 wait complete. 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient: build_initial_monmap 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient: monmap: 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:37.304 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 Processor -- start 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 -- start start 2024-09-16T07:18:37.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient: init 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea809e0b0 0x7f7ea81ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient(hunting): picked mon.noname-b con 0x7f7ea809e0b0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80a0ab0 0x7f7ea81342d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient(hunting): picked mon.noname-a con 0x7f7ea80a0ab0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ea8135810 0x7f7ea8134810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient(hunting): picked mon.noname-c con 0x7f7ea8135810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea81ca730 con 0x7f7ea80a0ab0 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea8125430 con 0x7f7ea809e0b0 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80a4170 con 0x7f7ea8135810 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient(hunting): authenticate will time out at 832.969546s 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea809e0b0 0x7f7ea81ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea809e0b0 0x7f7ea81ca1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45212/0 (socket says 172.21.15.143:45212) 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 1 -- 172.21.15.143:0/1957284763 learned_addr learned my addr 172.21.15.143:0/1957284763 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 10 monclient(hunting): get_auth_request con 0x7f7ea809e0b0 auth_method 0 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7ead20d700 1 --2- 172.21.15.143:0/1957284763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80a0ab0 0x7f7ea81342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eae20f700 1 --2- 172.21.15.143:0/1957284763 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ea8135810 0x7f7ea8134810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7ead20d700 10 monclient(hunting): get_auth_request con 0x7f7ea80a0ab0 auth_method 0 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7ead20d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7ead20d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7ead20d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eae20f700 10 monclient(hunting): get_auth_request con 0x7f7ea8135810 auth_method 0 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eae20f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eae20f700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eae20f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 1 -- 172.21.15.143:0/1957284763 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ea8135810 msgr2=0x7f7ea8134810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 1 --2- 172.21.15.143:0/1957284763 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ea8135810 0x7f7ea8134810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 1 -- 172.21.15.143:0/1957284763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80a0ab0 msgr2=0x7f7ea81342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 1 --2- 172.21.15.143:0/1957284763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80a0ab0 0x7f7ea81342d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 10 monclient: _finish_hunting 0 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 1 monclient: found mon.noname-b 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 1 -- 172.21.15.143:0/1957284763 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e98009570 con 0x7f7ea809e0b0 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eada0e700 1 --2- 172.21.15.143:0/1957284763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea809e0b0 0x7f7ea81ca1f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8067750 tx=0x7f7e98004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:37.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 1 -- 172.21.15.143:0/1957284763 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e980045d0 con 0x7f7ea809e0b0 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 10 monclient: dump: 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.310 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 10 monclient: _finish_auth 0 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 5 monclient: authenticate success, global_id 4120 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 1 -- 172.21.15.143:0/1957284763 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e980036c0 con 0x7f7ea809e0b0 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient: _renew_subs 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 1 -- 172.21.15.143:0/1957284763 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98003a50 con 0x7f7ea809e0b0 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 -- 172.21.15.143:0/1957284763 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea80706e0 con 0x7f7ea809e0b0 2024-09-16T07:18:37.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9f7fe700 4 set_mon_vals no callback set 2024-09-16T07:18:37.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:37.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 10 monclient: dump: 2024-09-16T07:18:37.312 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.312 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.312 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.312 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.312 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.313 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.313 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.313 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.313 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.313 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient: _renew_subs 2024-09-16T07:18:37.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 1 -- 172.21.15.143:0/1957284763 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea8125cd0 con 0x7f7ea809e0b0 2024-09-16T07:18:37.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 1 -- 172.21.15.143:0/1957284763 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f7e98021070 con 0x7f7ea809e0b0 2024-09-16T07:18:37.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7e9e7fc700 1 -- 172.21.15.143:0/1957284763 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f7e9801b7e0 con 0x7f7ea809e0b0 2024-09-16T07:18:37.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.290+0000 7f7eaea10700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.294+0000 7f7eaea10700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.294+0000 7f7eaea10700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.315 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.294+0000 7f7eaea10700 1 -- 172.21.15.143:0/1957284763 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea8135bf0 con 0x7f7ea809e0b0 2024-09-16T07:18:37.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.294+0000 7f7e9e7fc700 1 -- 172.21.15.143:0/1957284763 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f7e98029020 con 0x7f7ea809e0b0 2024-09-16T07:18:37.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.294+0000 7f7e9e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.316 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.294+0000 7f7e9e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:37.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.374+0000 7f7eaea10700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "52d577a5-08b7-452d-aa35-e1560215a09e", "id": 1}] 2024-09-16T07:18:37.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.374+0000 7f7eaea10700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "52d577a5-08b7-452d-aa35-e1560215a09e", "id": 1}] 2024-09-16T07:18:37.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.374+0000 7f7eaea10700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.374+0000 7f7eaea10700 1 -- 172.21.15.143:0/1957284763 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "52d577a5-08b7-452d-aa35-e1560215a09e", "id": 1} v 0) v1 -- 0x7f7ea80a1230 con 0x7f7ea809e0b0 2024-09-16T07:18:37.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7e9e7fc700 1 -- 172.21.15.143:0/1957284763 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "52d577a5-08b7-452d-aa35-e1560215a09e", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7e98015020 con 0x7f7ea809e0b0 2024-09-16T07:18:37.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7e9e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "52d577a5-08b7-452d-aa35-e1560215a09e", "id": 1}] 2024-09-16T07:18:37.381 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7e9e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:37.381 INFO:teuthology.orchestra.run.smithi143.stdout:1 2024-09-16T07:18:37.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 10 monclient: shutdown 2024-09-16T07:18:37.382 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:37.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 1 -- 172.21.15.143:0/1957284763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea809e0b0 msgr2=0x7f7ea81ca1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 1 --2- 172.21.15.143:0/1957284763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea809e0b0 0x7f7ea81ca1f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8067750 tx=0x7f7e98004770 comp rx=0 tx=0).stop 2024-09-16T07:18:37.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 1 -- 172.21.15.143:0/1957284763 shutdown_connections 2024-09-16T07:18:37.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 1 --2- 172.21.15.143:0/1957284763 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ea8135810 0x7f7ea8134810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 1 --2- 172.21.15.143:0/1957284763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80a0ab0 0x7f7ea81342d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 1 --2- 172.21.15.143:0/1957284763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea809e0b0 0x7f7ea81ca1f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 1 -- 172.21.15.143:0/1957284763 >> 172.21.15.143:0/1957284763 conn(0x7f7ea80a14c0 msgr2=0x7f7ea806f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:37.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 1 -- 172.21.15.143:0/1957284763 shutdown_connections 2024-09-16T07:18:37.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.378+0000 7f7eaea10700 1 -- 172.21.15.143:0/1957284763 wait complete. 2024-09-16T07:18:37.395 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new e3deb985-fb4e-42ee-8ed6-e6318cbca322 2 2024-09-16T07:18:37.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient: get_monmap_and_config 2024-09-16T07:18:37.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient: build_initial_monmap 2024-09-16T07:18:37.509 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient: monmap: 2024-09-16T07:18:37.509 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:37.509 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.509 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:37.509658+0000 2024-09-16T07:18:37.509 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:37.509658+0000 2024-09-16T07:18:37.509 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:37.510 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.510 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:37.510 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:37.510 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:37.510 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:37.510 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:37.510 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 1 Processor -- start 2024-09-16T07:18:37.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 1 -- start start 2024-09-16T07:18:37.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient: init 2024-09-16T07:18:37.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb1200a0e90 0x7fb1200a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient(hunting): picked mon.noname-c con 0x7fb1200a0e90 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1200a1270 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1200a2d50 0x7fb1200a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient(hunting): picked mon.noname-b con 0x7fb1200a2d50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1200a36c0 con 0x7fb1200a1270 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb12006cf50 con 0x7fb1200a0e90 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12009f810 con 0x7fb1200a2d50 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:37.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb126fac700 10 monclient(hunting): authenticate will time out at 833.187553s 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb125faa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb1200a0e90 0x7fb1200a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb125faa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb1200a0e90 0x7fb1200a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45224/0 (socket says 172.21.15.143:45224) 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb125faa700 1 -- 172.21.15.143:0/779891842 learned_addr learned my addr 172.21.15.143:0/779891842 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb125faa700 10 monclient(hunting): get_auth_request con 0x7fb1200a0e90 auth_method 0 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb125faa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb125faa700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb125faa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb125faa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb125faa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb125faa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb1267ab700 1 --2- 172.21.15.143:0/779891842 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1200a2d50 0x7fb1200a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb1267ab700 10 monclient(hunting): get_auth_request con 0x7fb1200a2d50 auth_method 0 2024-09-16T07:18:37.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb1267ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb1267ab700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.506+0000 7fb1267ab700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 -- 172.21.15.143:0/779891842 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1200a2d50 msgr2=0x7fb1200a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 --2- 172.21.15.143:0/779891842 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1200a2d50 0x7fb1200a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 -- 172.21.15.143:0/779891842 >> v1:172.21.15.37:6789/0 conn(0x7fb1200a1270 legacy=0x7fb1200a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient: _finish_hunting 0 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 monclient: found mon.noname-c 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 -- 172.21.15.143:0/779891842 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12009f250 con 0x7fb1200a0e90 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 --2- 172.21.15.143:0/779891842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 0x7fb1200a04d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb118009e50 tx=0x7fb11800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39de20016496c397 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:37.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 1 -- 172.21.15.143:0/779891842 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11800e070 con 0x7fb1200a0e90 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 10 monclient: dump: 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.524 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 10 monclient: _finish_auth 0 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 10 monclient: _check_auth_tickets 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 1 -- 172.21.15.143:0/779891842 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb118003a40 con 0x7fb1200a0e90 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 5 monclient: authenticate success, global_id 4123 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 4 set_mon_vals no callback set 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 1 -- 172.21.15.143:0/779891842 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118003df0 con 0x7fb1200a0e90 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.525 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb124fa8700 10 monclient: dump: 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient: shutdown 2024-09-16T07:18:37.526 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:37.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 -- 172.21.15.143:0/779891842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 msgr2=0x7fb1200a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 --2- 172.21.15.143:0/779891842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 0x7fb1200a04d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb118009e50 tx=0x7fb11800b330 comp rx=0 tx=0).stop 2024-09-16T07:18:37.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 -- 172.21.15.143:0/779891842 shutdown_connections 2024-09-16T07:18:37.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 --2- 172.21.15.143:0/779891842 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1200a2d50 0x7fb1200a3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 --2- 172.21.15.143:0/779891842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 0x7fb1200a04d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 -- 172.21.15.143:0/779891842 >> 172.21.15.143:0/779891842 conn(0x7fb1200a4740 msgr2=0x7fb1200a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:37.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 -- 172.21.15.143:0/779891842 shutdown_connections 2024-09-16T07:18:37.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 -- 172.21.15.143:0/779891842 wait complete. 2024-09-16T07:18:37.527 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient: build_initial_monmap 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient: monmap: 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 Processor -- start 2024-09-16T07:18:37.528 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 -- start start 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient: init 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 0x7fb12013bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient(hunting): picked mon.noname-b con 0x7fb1200a0e90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1200a1270 0x7fb12013c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient(hunting): picked mon.noname-c con 0x7fb1200a1270 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1200a2d50 0x7fb120122d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient(hunting): picked mon.noname-a con 0x7fb1200a2d50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb120126cc0 con 0x7fb1200a2d50 2024-09-16T07:18:37.529 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12006c060 con 0x7fb1200a0e90 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12006d780 con 0x7fb1200a1270 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb126fac700 10 monclient(hunting): authenticate will time out at 833.191225s 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 0x7fb12013bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 0x7fb12013bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45228/0 (socket says 172.21.15.143:45228) 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 -- 172.21.15.143:0/3760759375 learned_addr learned my addr 172.21.15.143:0/3760759375 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient(hunting): get_auth_request con 0x7fb1200a0e90 auth_method 0 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.530 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb1267ab700 1 --2- 172.21.15.143:0/3760759375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1200a2d50 0x7fb120122d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb1257a9700 1 --2- 172.21.15.143:0/3760759375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1200a1270 0x7fb12013c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb1267ab700 10 monclient(hunting): get_auth_request con 0x7fb1200a2d50 auth_method 0 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb1267ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb1267ab700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb1267ab700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb1257a9700 10 monclient(hunting): get_auth_request con 0x7fb1200a1270 auth_method 0 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb1257a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb1257a9700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb1257a9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-16T07:18:37.531 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 -- 172.21.15.143:0/3760759375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1200a1270 msgr2=0x7fb12013c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 --2- 172.21.15.143:0/3760759375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1200a1270 0x7fb12013c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 -- 172.21.15.143:0/3760759375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1200a2d50 msgr2=0x7fb120122d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 --2- 172.21.15.143:0/3760759375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1200a2d50 0x7fb120122d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient: _finish_hunting 0 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 monclient: found mon.noname-b 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 -- 172.21.15.143:0/3760759375 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118009b30 con 0x7fb1200a0e90 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.510+0000 7fb125faa700 1 --2- 172.21.15.143:0/3760759375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 0x7fb12013bd70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb118015040 tx=0x7fb118004af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 1 -- 172.21.15.143:0/3760759375 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118004440 con 0x7fb1200a0e90 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:37.532 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 10 monclient: dump: 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 10 monclient: _finish_auth 0 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 10 monclient: _check_auth_tickets 2024-09-16T07:18:37.533 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 5 monclient: authenticate success, global_id 4126 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 1 -- 172.21.15.143:0/3760759375 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1180045a0 con 0x7fb1200a0e90 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 10 monclient: _renew_subs 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 1 -- 172.21.15.143:0/3760759375 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12009e010 con 0x7fb1200a0e90 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb117fff700 4 set_mon_vals no callback set 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 10 monclient: _renew_subs 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 1 -- 172.21.15.143:0/3760759375 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118004950 con 0x7fb1200a0e90 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 1 -- 172.21.15.143:0/3760759375 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1201269e0 con 0x7fb1200a0e90 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:37.534 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 10 monclient: dump: 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 1 -- 172.21.15.143:0/3760759375 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fb118025750 con 0x7fb1200a0e90 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb1167fc700 1 -- 172.21.15.143:0/3760759375 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fb11801c400 con 0x7fb1200a0e90 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.535 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.514+0000 7fb126fac700 1 -- 172.21.15.143:0/3760759375 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1200a1680 con 0x7fb1200a0e90 2024-09-16T07:18:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.518+0000 7fb1167fc700 1 -- 172.21.15.143:0/3760759375 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fb1200a1680 con 0x7fb1200a0e90 2024-09-16T07:18:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.518+0000 7fb1167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.536 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.518+0000 7fb1167fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:37.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.598+0000 7fb126fac700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e3deb985-fb4e-42ee-8ed6-e6318cbca322", "id": 2}] 2024-09-16T07:18:37.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.598+0000 7fb126fac700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e3deb985-fb4e-42ee-8ed6-e6318cbca322", "id": 2}] 2024-09-16T07:18:37.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.598+0000 7fb126fac700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.599 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.598+0000 7fb126fac700 1 -- 172.21.15.143:0/3760759375 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e3deb985-fb4e-42ee-8ed6-e6318cbca322", "id": 2} v 0) v1 -- 0x7fb12009f560 con 0x7fb1200a0e90 2024-09-16T07:18:37.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb1167fc700 1 -- 172.21.15.143:0/3760759375 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e3deb985-fb4e-42ee-8ed6-e6318cbca322", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb118020070 con 0x7fb1200a0e90 2024-09-16T07:18:37.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb1167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e3deb985-fb4e-42ee-8ed6-e6318cbca322", "id": 2}] 2024-09-16T07:18:37.605 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb1167fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:37.605 INFO:teuthology.orchestra.run.smithi143.stdout:2 2024-09-16T07:18:37.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb126fac700 10 monclient: shutdown 2024-09-16T07:18:37.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb126fac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:37.606 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb126fac700 1 -- 172.21.15.143:0/3760759375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 msgr2=0x7fb12013bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb126fac700 1 --2- 172.21.15.143:0/3760759375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 0x7fb12013bd70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb118015040 tx=0x7fb118004af0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb126fac700 1 -- 172.21.15.143:0/3760759375 shutdown_connections 2024-09-16T07:18:37.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb126fac700 1 --2- 172.21.15.143:0/3760759375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1200a2d50 0x7fb120122d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb126fac700 1 --2- 172.21.15.143:0/3760759375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1200a1270 0x7fb12013c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb126fac700 1 --2- 172.21.15.143:0/3760759375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1200a0e90 0x7fb12013bd70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb126fac700 1 -- 172.21.15.143:0/3760759375 >> 172.21.15.143:0/3760759375 conn(0x7fb1200a4740 msgr2=0x7fb1200a71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:37.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.602+0000 7fb126fac700 1 -- 172.21.15.143:0/3760759375 shutdown_connections 2024-09-16T07:18:37.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.606+0000 7fb126fac700 1 -- 172.21.15.143:0/3760759375 wait complete. 2024-09-16T07:18:37.619 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new ab447540-8ed1-4a0c-8f6c-dfeea8af02ef 3 2024-09-16T07:18:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient: get_monmap_and_config 2024-09-16T07:18:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient: build_initial_monmap 2024-09-16T07:18:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient: monmap: 2024-09-16T07:18:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:37.731729+0000 2024-09-16T07:18:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:37.731729+0000 2024-09-16T07:18:37.731 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:37.732 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.732 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:37.732 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:37.732 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:37.732 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:37.732 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:37.732 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.736 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 1 Processor -- start 2024-09-16T07:18:37.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 1 -- start start 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient: init 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a0e90 0x7f49f00a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient(hunting): picked mon.noname-b con 0x7f49f00a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49f00a1270 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f49f00a2d50 0x7f49f00a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient(hunting): picked mon.noname-a con 0x7f49f00a2d50 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f00a36c0 con 0x7f49f00a1270 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f006cf50 con 0x7f49f00a2d50 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f009f810 con 0x7f49f00a0e90 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient(hunting): authenticate will time out at 833.409529s 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ee7fc700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f49f00a1270 0x7f49f00a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:50968/0 (socket says 172.21.15.143:50968) 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ee7fc700 1 -- 172.21.15.143:0/53159508 learned_addr learned my addr 172.21.15.143:0/53159508 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ef7fe700 1 --2- 172.21.15.143:0/53159508 >> v2:172.21.15.37:3300/0 conn(0x7f49f00a2d50 0x7f49f00a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 1 --2- 172.21.15.143:0/53159508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a0e90 0x7f49f00a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ef7fe700 10 monclient(hunting): get_auth_request con 0x7f49f00a2d50 auth_method 0 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 10 monclient(hunting): get_auth_request con 0x7f49f00a0e90 auth_method 0 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ef7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ef7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ef7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 1 -- 172.21.15.143:0/53159508 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037430 0 0) 0x7f49f00a36c0 con 0x7f49f00a1270 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient(hunting): _negotiate 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.740 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient(hunting): my global_id is 4129 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 1 -- 172.21.15.143:0/53159508 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d80035d0 con 0x7f49f00a1270 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 1 -- 172.21.15.143:0/53159508 >> v2:172.21.15.37:3300/0 conn(0x7f49f00a2d50 msgr2=0x7f49f00a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 1 --2- 172.21.15.143:0/53159508 >> v2:172.21.15.37:3300/0 conn(0x7f49f00a2d50 0x7f49f00a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 1 -- 172.21.15.143:0/53159508 >> v1:172.21.15.143:6789/0 conn(0x7f49f00a1270 legacy=0x7f49f00a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 10 monclient: _finish_hunting 0 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 1 monclient: found mon.noname-b 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:18:37.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 1 -- 172.21.15.143:0/53159508 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f009f250 con 0x7f49f00a0e90 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49ef7fe700 1 --2- 172.21.15.143:0/53159508 >> v2:172.21.15.37:3300/0 conn(0x7f49f00a2d50 0x7f49f00a3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49eeffd700 1 --2- 172.21.15.143:0/53159508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a0e90 0x7f49f00a04d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009850 tx=0x7f49e8009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad434ba585517c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 1 -- 172.21.15.143:0/53159508 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8011830 con 0x7f49f00a0e90 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient: dump: 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.742 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient: _finish_auth 0 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient: _check_auth_tickets 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 1 -- 172.21.15.143:0/53159508 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e8011990 con 0x7f49f00a0e90 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 1 -- 172.21.15.143:0/53159508 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8011d40 con 0x7f49f00a0e90 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 5 monclient: authenticate success, global_id 4118 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49effff700 4 set_mon_vals no callback set 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.743 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.730+0000 7f49edffb700 10 monclient: dump: 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient: shutdown 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:37.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 -- 172.21.15.143:0/53159508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a0e90 msgr2=0x7f49f00a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 --2- 172.21.15.143:0/53159508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a0e90 0x7f49f00a04d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009850 tx=0x7f49e8009b60 comp rx=0 tx=0).stop 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 -- 172.21.15.143:0/53159508 shutdown_connections 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 --2- 172.21.15.143:0/53159508 >> v2:172.21.15.37:3300/0 conn(0x7f49f00a2d50 0x7f49f00a3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 --2- 172.21.15.143:0/53159508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a0e90 0x7f49f00a04d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 -- 172.21.15.143:0/53159508 >> 172.21.15.143:0/53159508 conn(0x7f49f00a4740 msgr2=0x7f49f00a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 -- 172.21.15.143:0/53159508 shutdown_connections 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 -- 172.21.15.143:0/53159508 wait complete. 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient: build_initial_monmap 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient: monmap: 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T07:18:37.745 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:37.746 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T07:18:37.746 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:37.746 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:37.746 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:37.746 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 Processor -- start 2024-09-16T07:18:37.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 -- start start 2024-09-16T07:18:37.746 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient: init 2024-09-16T07:18:37.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49f00a0e90 0x7f49f01ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient(hunting): picked mon.noname-b con 0x7f49f00a0e90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a1270 0x7f49f01caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient(hunting): picked mon.noname-c con 0x7f49f00a1270 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49f00a2d50 0x7f49f0122ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient(hunting): picked mon.noname-a con 0x7f49f00a2d50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f006d780 con 0x7f49f00a2d50 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f01266d0 con 0x7f49f00a0e90 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f0125220 con 0x7f49f00a1270 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49effff700 10 monclient(hunting): authenticate will time out at 833.415328s 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49eeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49f00a0e90 0x7f49f01ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49eeffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49f00a0e90 0x7f49f01ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45234/0 (socket says 172.21.15.143:45234) 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49eeffd700 1 -- 172.21.15.143:0/793756240 learned_addr learned my addr 172.21.15.143:0/793756240 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49eeffd700 10 monclient(hunting): get_auth_request con 0x7f49f00a0e90 auth_method 0 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49eeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49eeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49eeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49eeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ef7fe700 1 --2- 172.21.15.143:0/793756240 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49f00a2d50 0x7f49f0122ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ef7fe700 10 monclient(hunting): get_auth_request con 0x7f49f00a2d50 auth_method 0 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ee7fc700 1 --2- 172.21.15.143:0/793756240 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a1270 0x7f49f01caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ee7fc700 10 monclient(hunting): get_auth_request con 0x7f49f00a1270 auth_method 0 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ee7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ef7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ef7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ef7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ee7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ee7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.734+0000 7f49ee7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-16T07:18:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 1 -- 172.21.15.143:0/793756240 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a1270 msgr2=0x7f49f01caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 1 --2- 172.21.15.143:0/793756240 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a1270 0x7f49f01caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 1 -- 172.21.15.143:0/793756240 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49f00a2d50 msgr2=0x7f49f0122ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 1 --2- 172.21.15.143:0/793756240 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49f00a2d50 0x7f49f0122ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 10 monclient: _finish_hunting 0 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 1 monclient: found mon.noname-b 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 1 -- 172.21.15.143:0/793756240 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e0009a70 con 0x7f49f00a0e90 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49eeffd700 1 --2- 172.21.15.143:0/793756240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49f00a0e90 0x7f49f01ca4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49e80090b0 tx=0x7f49e8004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 1 -- 172.21.15.143:0/793756240 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e80042c0 con 0x7f49f00a0e90 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 10 monclient: dump: 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.755 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 10 monclient: _finish_auth 0 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 10 monclient: _check_auth_tickets 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 5 monclient: authenticate success, global_id 4132 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 1 -- 172.21.15.143:0/793756240 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e801f070 con 0x7f49f00a0e90 2024-09-16T07:18:37.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 10 monclient: _renew_subs 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 1 -- 172.21.15.143:0/793756240 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e80036c0 con 0x7f49f00a0e90 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 1 -- 172.21.15.143:0/793756240 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e80094b0 con 0x7f49f00a0e90 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 10 monclient: dump: 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.757 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49ed7fa700 4 set_mon_vals no callback set 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 10 monclient: _renew_subs 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 1 -- 172.21.15.143:0/793756240 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f0124650 con 0x7f49f00a0e90 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 1 -- 172.21.15.143:0/793756240 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f49e8021030 con 0x7f49f00a0e90 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49d77fe700 1 -- 172.21.15.143:0/793756240 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f49e801ad40 con 0x7f49f00a0e90 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.738+0000 7f49effff700 1 -- 172.21.15.143:0/793756240 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f00a04d0 con 0x7f49f00a0e90 2024-09-16T07:18:37.758 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.742+0000 7f49d77fe700 1 -- 172.21.15.143:0/793756240 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f49e8015020 con 0x7f49f00a0e90 2024-09-16T07:18:37.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.742+0000 7f49d77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.759 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.742+0000 7f49d77fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:37.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.814+0000 7f49effff700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ab447540-8ed1-4a0c-8f6c-dfeea8af02ef", "id": 3}] 2024-09-16T07:18:37.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.814+0000 7f49effff700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ab447540-8ed1-4a0c-8f6c-dfeea8af02ef", "id": 3}] 2024-09-16T07:18:37.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.814+0000 7f49effff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.819 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.814+0000 7f49effff700 1 -- 172.21.15.143:0/793756240 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ab447540-8ed1-4a0c-8f6c-dfeea8af02ef", "id": 3} v 0) v1 -- 0x7f49f00a2110 con 0x7f49f00a0e90 2024-09-16T07:18:37.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49d77fe700 1 -- 172.21.15.143:0/793756240 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ab447540-8ed1-4a0c-8f6c-dfeea8af02ef", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f49e801a450 con 0x7f49f00a0e90 2024-09-16T07:18:37.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49d77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ab447540-8ed1-4a0c-8f6c-dfeea8af02ef", "id": 3}] 2024-09-16T07:18:37.825 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49d77fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:37.825 INFO:teuthology.orchestra.run.smithi143.stdout:3 2024-09-16T07:18:37.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 10 monclient: shutdown 2024-09-16T07:18:37.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:37.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 1 -- 172.21.15.143:0/793756240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49f00a0e90 msgr2=0x7f49f01ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 1 --2- 172.21.15.143:0/793756240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49f00a0e90 0x7f49f01ca4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f49e80090b0 tx=0x7f49e8004770 comp rx=0 tx=0).stop 2024-09-16T07:18:37.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 1 -- 172.21.15.143:0/793756240 shutdown_connections 2024-09-16T07:18:37.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 1 --2- 172.21.15.143:0/793756240 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f49f00a2d50 0x7f49f0122ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.826 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 1 --2- 172.21.15.143:0/793756240 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f49f00a1270 0x7f49f01caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 1 --2- 172.21.15.143:0/793756240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49f00a0e90 0x7f49f01ca4e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 1 -- 172.21.15.143:0/793756240 >> 172.21.15.143:0/793756240 conn(0x7f49f00a4740 msgr2=0x7f49f009e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:37.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 1 -- 172.21.15.143:0/793756240 shutdown_connections 2024-09-16T07:18:37.827 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.822+0000 7f49effff700 1 -- 172.21.15.143:0/793756240 wait complete. 2024-09-16T07:18:37.839 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new be421fc0-1edf-4f20-b530-c9cf4eb20a99 4 2024-09-16T07:18:37.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient: get_monmap_and_config 2024-09-16T07:18:37.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient: build_initial_monmap 2024-09-16T07:18:37.947 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient: monmap: 2024-09-16T07:18:37.947 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:37.947 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.947 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:37.947960+0000 2024-09-16T07:18:37.947 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:37.947960+0000 2024-09-16T07:18:37.948 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:37.948 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.948 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:37.948 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:37.948 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:37.948 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:37.948 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:37.948 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 Processor -- start 2024-09-16T07:18:37.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 -- start start 2024-09-16T07:18:37.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient: init 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f242c0a0e90 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f242c0a04d0 0x7f242c0a1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient(hunting): picked mon.noname-b con 0x7f242c0a04d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f242c0a2d50 0x7f242c0a31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient(hunting): picked mon.noname-c con 0x7f242c0a2d50 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242c0a0900 con 0x7f242c0a0e90 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f242c0a17b0 con 0x7f242c0a2d50 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c06cf50 con 0x7f242c0a04d0 2024-09-16T07:18:37.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient(hunting): authenticate will time out at 833.625721s 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f242c0a2d50 0x7f242c0a31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f242c0a2d50 0x7f242c0a31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45238/0 (socket says 172.21.15.143:45238) 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 1 -- 172.21.15.143:0/3624908450 learned_addr learned my addr 172.21.15.143:0/3624908450 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 10 monclient(hunting): get_auth_request con 0x7f242c0a2d50 auth_method 0 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2430bc4700 1 --2- 172.21.15.143:0/3624908450 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f242c0a04d0 0x7f242c0a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2430bc4700 10 monclient(hunting): get_auth_request con 0x7f242c0a04d0 auth_method 0 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2430bc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2430bc4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2430bc4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.958 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2430bc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2430bc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2430bc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 1 -- 172.21.15.143:0/3624908450 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f242c0a04d0 msgr2=0x7f242c0a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 1 --2- 172.21.15.143:0/3624908450 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f242c0a04d0 0x7f242c0a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 1 -- 172.21.15.143:0/3624908450 >> v1:172.21.15.37:6789/0 conn(0x7f242c0a0e90 legacy=0x7f242c0a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 10 monclient: _finish_hunting 0 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 1 monclient: found mon.noname-c 2024-09-16T07:18:37.959 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 1 -- 172.21.15.143:0/3624908450 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c0a36e0 con 0x7f242c0a2d50 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f2431bc6700 1 --2- 172.21.15.143:0/3624908450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 0x7f242c0a31a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f241c009910 tx=0x7f241c009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3197d787092f184 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 1 -- 172.21.15.143:0/3624908450 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c011830 con 0x7f242c0a2d50 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 10 monclient: dump: 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.960 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 10 monclient: _finish_auth 0 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 10 monclient: _check_auth_tickets 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 1 -- 172.21.15.143:0/3624908450 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241c011990 con 0x7f242c0a2d50 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 5 monclient: authenticate success, global_id 4135 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 4 set_mon_vals no callback set 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 1 -- 172.21.15.143:0/3624908450 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c011d40 con 0x7f242c0a2d50 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f242bfff700 10 monclient: dump: 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 10 monclient: shutdown 2024-09-16T07:18:37.962 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 -- 172.21.15.143:0/3624908450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 msgr2=0x7f242c0a31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 --2- 172.21.15.143:0/3624908450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 0x7f242c0a31a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f241c009910 tx=0x7f241c009c20 comp rx=0 tx=0).stop 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 -- 172.21.15.143:0/3624908450 shutdown_connections 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 --2- 172.21.15.143:0/3624908450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 0x7f242c0a31a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 --2- 172.21.15.143:0/3624908450 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f242c0a04d0 0x7f242c0a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.946+0000 7f24323c7700 1 -- 172.21.15.143:0/3624908450 >> 172.21.15.143:0/3624908450 conn(0x7f242c0a4740 msgr2=0x7f242c0a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 -- 172.21.15.143:0/3624908450 shutdown_connections 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 -- 172.21.15.143:0/3624908450 wait complete. 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient: build_initial_monmap 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient: monmap: 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:37.963 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 Processor -- start 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 -- start start 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient: init 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:18:37.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f242c0a04d0 0x7f242c1ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient(hunting): picked mon.noname-c con 0x7f242c0a04d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f242c0a0e90 0x7f242c1caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient(hunting): picked mon.noname-a con 0x7f242c0a0e90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 0x7f242c122aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient(hunting): picked mon.noname-b con 0x7f242c0a2d50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c06d780 con 0x7f242c0a0e90 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c1266e0 con 0x7f242c0a2d50 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242c1269c0 con 0x7f242c0a04d0 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:37.965 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient(hunting): authenticate will time out at 833.629966s 2024-09-16T07:18:37.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 0x7f242c122aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 0x7f242c122aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45252/0 (socket says 172.21.15.143:45252) 2024-09-16T07:18:37.966 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 1 -- 172.21.15.143:0/2100432178 learned_addr learned my addr 172.21.15.143:0/2100432178 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T07:18:37.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 10 monclient(hunting): get_auth_request con 0x7f242c0a2d50 auth_method 0 2024-09-16T07:18:37.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24313c5700 1 --2- 172.21.15.143:0/2100432178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f242c0a04d0 0x7f242c1ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24313c5700 10 monclient(hunting): get_auth_request con 0x7f242c0a04d0 auth_method 0 2024-09-16T07:18:37.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24313c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24313c5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24313c5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2430bc4700 1 --2- 172.21.15.143:0/2100432178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f242c0a0e90 0x7f242c1caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:37.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2430bc4700 10 monclient(hunting): get_auth_request con 0x7f242c0a0e90 auth_method 0 2024-09-16T07:18:37.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2430bc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:37.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2430bc4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:37.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2430bc4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:37.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2430bc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.968 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2430bc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2430bc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24313c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:37.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24313c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:37.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24313c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:37.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-16T07:18:37.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 1 -- 172.21.15.143:0/2100432178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f242c0a04d0 msgr2=0x7f242c1ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 1 --2- 172.21.15.143:0/2100432178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f242c0a04d0 0x7f242c1ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 1 -- 172.21.15.143:0/2100432178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f242c0a0e90 msgr2=0x7f242c1caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:37.969 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 1 --2- 172.21.15.143:0/2100432178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f242c0a0e90 0x7f242c1caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 10 monclient: _finish_hunting 0 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 1 monclient: found mon.noname-b 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 1 -- 172.21.15.143:0/2100432178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c009570 con 0x7f242c0a2d50 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2431bc6700 1 --2- 172.21.15.143:0/2100432178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 0x7f242c122aa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f241c009f70 tx=0x7f241c004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 1 -- 172.21.15.143:0/2100432178 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c004300 con 0x7f242c0a2d50 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 10 monclient: dump: 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.970 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 10 monclient: _finish_auth 0 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 5 monclient: authenticate success, global_id 4138 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 1 -- 172.21.15.143:0/2100432178 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241c01f070 con 0x7f242c0a2d50 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient: _renew_subs 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.971 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 -- 172.21.15.143:0/2100432178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c09e000 con 0x7f242c0a2d50 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 1 -- 172.21.15.143:0/2100432178 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c0036c0 con 0x7f242c0a2d50 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 10 monclient: dump: 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:37.972 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient: _renew_subs 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f24323c7700 1 -- 172.21.15.143:0/2100432178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c06bf90 con 0x7f242c0a2d50 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f242affd700 4 set_mon_vals no callback set 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 1 -- 172.21.15.143:0/2100432178 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f241c021030 con 0x7f242c0a2d50 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.950+0000 7f2429ffb700 1 -- 172.21.15.143:0/2100432178 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f241c006d00 con 0x7f242c0a2d50 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.954+0000 7f24323c7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.954+0000 7f24323c7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.954+0000 7f24323c7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:37.973 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.954+0000 7f24323c7700 1 -- 172.21.15.143:0/2100432178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c126520 con 0x7f242c0a2d50 2024-09-16T07:18:37.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.958+0000 7f2429ffb700 1 -- 172.21.15.143:0/2100432178 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f241c044e30 con 0x7f242c0a2d50 2024-09-16T07:18:37.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.958+0000 7f2429ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:37.974 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:37.958+0000 7f2429ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:38.050 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:38.046+0000 7f8b087a32c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T07:18:38.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.050+0000 7f24323c7700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "be421fc0-1edf-4f20-b530-c9cf4eb20a99", "id": 4}] 2024-09-16T07:18:38.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.050+0000 7f24323c7700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "be421fc0-1edf-4f20-b530-c9cf4eb20a99", "id": 4}] 2024-09-16T07:18:38.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.050+0000 7f24323c7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:38.051 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.050+0000 7f24323c7700 1 -- 172.21.15.143:0/2100432178 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "be421fc0-1edf-4f20-b530-c9cf4eb20a99", "id": 4} v 0) v1 -- 0x7f242c125230 con 0x7f242c0a2d50 2024-09-16T07:18:38.058 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.054+0000 7f2429ffb700 1 -- 172.21.15.143:0/2100432178 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be421fc0-1edf-4f20-b530-c9cf4eb20a99", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f241c033070 con 0x7f242c0a2d50 2024-09-16T07:18:38.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f2429ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "be421fc0-1edf-4f20-b530-c9cf4eb20a99", "id": 4}] 2024-09-16T07:18:38.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f2429ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:38.059 INFO:teuthology.orchestra.run.smithi143.stdout:4 2024-09-16T07:18:38.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 10 monclient: shutdown 2024-09-16T07:18:38.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:38.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 1 -- 172.21.15.143:0/2100432178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 msgr2=0x7f242c122aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:38.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 1 --2- 172.21.15.143:0/2100432178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 0x7f242c122aa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f241c009f70 tx=0x7f241c004770 comp rx=0 tx=0).stop 2024-09-16T07:18:38.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 1 -- 172.21.15.143:0/2100432178 shutdown_connections 2024-09-16T07:18:38.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 1 --2- 172.21.15.143:0/2100432178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f242c0a2d50 0x7f242c122aa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 1 --2- 172.21.15.143:0/2100432178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f242c0a0e90 0x7f242c1caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 1 --2- 172.21.15.143:0/2100432178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f242c0a04d0 0x7f242c1ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 1 -- 172.21.15.143:0/2100432178 >> 172.21.15.143:0/2100432178 conn(0x7f242c0a4740 msgr2=0x7f242c09c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:38.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 1 -- 172.21.15.143:0/2100432178 shutdown_connections 2024-09-16T07:18:38.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.058+0000 7f24323c7700 1 -- 172.21.15.143:0/2100432178 wait complete. 2024-09-16T07:18:38.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 8c7725bd-5f56-4fa0-b455-6569e313f9fc 5 2024-09-16T07:18:38.110 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:38.106+0000 7f8b087a32c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T07:18:38.186 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.182+0000 7fc91b408700 10 monclient: get_monmap_and_config 2024-09-16T07:18:38.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.182+0000 7fc91b408700 10 monclient: build_initial_monmap 2024-09-16T07:18:38.187 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.182+0000 7fc91b408700 10 monclient: monmap: 2024-09-16T07:18:38.187 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:38.187 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:38.187 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:38.186969+0000 2024-09-16T07:18:38.187 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:38.186969+0000 2024-09-16T07:18:38.187 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:38.187 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:38.187 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:38.187 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:38.188 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:38.188 INFO:teuthology.orchestra.run.smithi143.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:38.188 INFO:teuthology.orchestra.run.smithi143.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:38.188 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:38.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 Processor -- start 2024-09-16T07:18:38.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 -- start start 2024-09-16T07:18:38.188 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient: init 2024-09-16T07:18:38.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:38.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T07:18:38.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc9140a0e90 0x7fc9140a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:38.196 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient(hunting): picked mon.noname-c con 0x7fc9140a0e90 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:38.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc9140a1270 0x7fc9140a2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:38.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient(hunting): picked mon.noname-a con 0x7fc9140a1270 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:38.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9140a2b80 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:38.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140a36c0 con 0x7fc9140a2b80 2024-09-16T07:18:38.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:38.197 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc91406cf50 con 0x7fc9140a1270 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc91409f810 con 0x7fc9140a0e90 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient(hunting): authenticate will time out at 833.864922s 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc9140a0e90 0x7fc9140a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc9140a0e90 0x7fc9140a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45262/0 (socket says 172.21.15.143:45262) 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 1 -- 172.21.15.143:0/4242880571 learned_addr learned my addr 172.21.15.143:0/4242880571 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 10 monclient(hunting): get_auth_request con 0x7fc9140a0e90 auth_method 0 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.200 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919c05700 1 --2- 172.21.15.143:0/4242880571 >> v2:172.21.15.37:3300/0 conn(0x7fc9140a1270 0x7fc9140a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919c05700 10 monclient(hunting): get_auth_request con 0x7fc9140a1270 auth_method 0 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919c05700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919c05700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919c05700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 1 -- 172.21.15.143:0/4242880571 >> v2:172.21.15.37:3300/0 conn(0x7fc9140a1270 msgr2=0x7fc9140a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 1 --2- 172.21.15.143:0/4242880571 >> v2:172.21.15.37:3300/0 conn(0x7fc9140a1270 0x7fc9140a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.201 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 1 -- 172.21.15.143:0/4242880571 >> v1:172.21.15.37:6789/0 conn(0x7fc9140a2b80 legacy=0x7fc9140a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 10 monclient: _finish_hunting 0 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 1 monclient: found mon.noname-c 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 1 -- 172.21.15.143:0/4242880571 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91409f250 con 0x7fc9140a0e90 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91a406700 1 --2- 172.21.15.143:0/4242880571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a0e90 0x7fc9140a04d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc914072af0 tx=0x7fc910009bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53b9093ee33945b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 1 -- 172.21.15.143:0/4242880571 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910004230 con 0x7fc9140a0e90 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 10 monclient: dump: 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:38.202 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 10 monclient: _finish_auth 0 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 10 monclient: _check_auth_tickets 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:38.203 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 1 -- 172.21.15.143:0/4242880571 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc910004390 con 0x7fc9140a0e90 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 5 monclient: authenticate success, global_id 4141 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 1 -- 172.21.15.143:0/4242880571 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9100036c0 con 0x7fc9140a0e90 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 4 set_mon_vals no callback set 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc919404700 10 monclient: dump: 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:38.204 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient: shutdown 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 -- 172.21.15.143:0/4242880571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a0e90 msgr2=0x7fc9140a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 --2- 172.21.15.143:0/4242880571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a0e90 0x7fc9140a04d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc914072af0 tx=0x7fc910009bf0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 -- 172.21.15.143:0/4242880571 shutdown_connections 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 --2- 172.21.15.143:0/4242880571 >> v2:172.21.15.37:3300/0 conn(0x7fc9140a1270 0x7fc9140a2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 --2- 172.21.15.143:0/4242880571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a0e90 0x7fc9140a04d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 -- 172.21.15.143:0/4242880571 >> 172.21.15.143:0/4242880571 conn(0x7fc9140a4740 msgr2=0x7fc9140a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 -- 172.21.15.143:0/4242880571 shutdown_connections 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 1 -- 172.21.15.143:0/4242880571 wait complete. 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient: build_initial_monmap 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.186+0000 7fc91b408700 10 monclient: monmap: 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-09-16T07:18:38.206 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 Processor -- start 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 -- start start 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient: init 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9140a0e90 0x7fc9141c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient(hunting): picked mon.noname-c con 0x7fc9140a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:38.207 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9140a1270 0x7fc9141c3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient(hunting): picked mon.noname-a con 0x7fc9140a1270 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a2b80 0x7fc9141c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient(hunting): picked mon.noname-b con 0x7fc9140a2b80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9141a8e90 con 0x7fc9140a1270 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91406bfa0 con 0x7fc9140a2b80 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91409dfc0 con 0x7fc9140a0e90 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:38.208 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient(hunting): authenticate will time out at 833.868757s 2024-09-16T07:18:38.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a2b80 0x7fc9141c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:38.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a2b80 0x7fc9141c6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45272/0 (socket says 172.21.15.143:45272) 2024-09-16T07:18:38.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 1 -- 172.21.15.143:0/3481065623 learned_addr learned my addr 172.21.15.143:0/3481065623 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-09-16T07:18:38.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 10 monclient(hunting): get_auth_request con 0x7fc9140a2b80 auth_method 0 2024-09-16T07:18:38.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:38.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:38.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:38.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91a406700 1 --2- 172.21.15.143:0/3481065623 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9140a0e90 0x7fc9141c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc919c05700 1 --2- 172.21.15.143:0/3481065623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9140a1270 0x7fc9141c3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc919c05700 10 monclient(hunting): get_auth_request con 0x7fc9140a1270 auth_method 0 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc919c05700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc919c05700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc919c05700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91a406700 10 monclient(hunting): get_auth_request con 0x7fc9140a0e90 auth_method 0 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91a406700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91a406700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91a406700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc919c05700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:38.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc919c05700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc919c05700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91a406700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91a406700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91a406700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 1 -- 172.21.15.143:0/3481065623 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9140a0e90 msgr2=0x7fc9141c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 1 --2- 172.21.15.143:0/3481065623 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9140a0e90 0x7fc9141c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 1 -- 172.21.15.143:0/3481065623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9140a1270 msgr2=0x7fc9141c3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 1 --2- 172.21.15.143:0/3481065623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9140a1270 0x7fc9141c3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 10 monclient: _finish_hunting 0 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 1 monclient: found mon.noname-b 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T07:18:38.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 1 -- 172.21.15.143:0/3481065623 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc910009570 con 0x7fc9140a2b80 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91ac07700 1 --2- 172.21.15.143:0/3481065623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a2b80 0x7fc9141c6a40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00cc10 tx=0x7fc90c00cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 1 -- 172.21.15.143:0/3481065623 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90c00aa60 con 0x7fc9140a2b80 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 10 monclient: dump: 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:38.212 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 10 monclient: _finish_auth 0 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 5 monclient: authenticate success, global_id 4144 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 1 -- 172.21.15.143:0/3481065623 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90c00abc0 con 0x7fc9140a2b80 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient: _renew_subs 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 -- 172.21.15.143:0/3481065623 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91406d780 con 0x7fc9140a2b80 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 1 -- 172.21.15.143:0/3481065623 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90c012420 con 0x7fc9140a2b80 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:38.213 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 10 monclient: dump: 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc918c03700 4 set_mon_vals no callback set 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient: _renew_subs 2024-09-16T07:18:38.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 -- 172.21.15.143:0/3481065623 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9141ab090 con 0x7fc9140a2b80 2024-09-16T07:18:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 1 -- 172.21.15.143:0/3481065623 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fc90c012880 con 0x7fc9140a2b80 2024-09-16T07:18:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc90affd700 1 -- 172.21.15.143:0/3481065623 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fc90c01c460 con 0x7fc9140a2b80 2024-09-16T07:18:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.190+0000 7fc91b408700 1 -- 172.21.15.143:0/3481065623 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9140a04d0 con 0x7fc9140a2b80 2024-09-16T07:18:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.194+0000 7fc90affd700 1 -- 172.21.15.143:0/3481065623 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fc90c012b30 con 0x7fc9140a2b80 2024-09-16T07:18:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.194+0000 7fc90affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:38.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.194+0000 7fc90affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:38.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.290+0000 7fc91b408700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8c7725bd-5f56-4fa0-b455-6569e313f9fc", "id": 5}] 2024-09-16T07:18:38.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.290+0000 7fc91b408700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8c7725bd-5f56-4fa0-b455-6569e313f9fc", "id": 5}] 2024-09-16T07:18:38.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.290+0000 7fc91b408700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:38.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.290+0000 7fc91b408700 1 -- 172.21.15.143:0/3481065623 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8c7725bd-5f56-4fa0-b455-6569e313f9fc", "id": 5} v 0) v1 -- 0x7fc91409f5c0 con 0x7fc9140a2b80 2024-09-16T07:18:38.295 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:38.294+0000 7f8b087a32c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T07:18:38.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc90affd700 1 -- 172.21.15.143:0/3481065623 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8c7725bd-5f56-4fa0-b455-6569e313f9fc", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc90c012cd0 con 0x7fc9140a2b80 2024-09-16T07:18:38.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc90affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8c7725bd-5f56-4fa0-b455-6569e313f9fc", "id": 5}] 2024-09-16T07:18:38.300 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc90affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:38.301 INFO:teuthology.orchestra.run.smithi143.stdout:5 2024-09-16T07:18:38.302 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 10 monclient: shutdown 2024-09-16T07:18:38.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:38.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 1 -- 172.21.15.143:0/3481065623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a2b80 msgr2=0x7fc9141c6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:38.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 1 --2- 172.21.15.143:0/3481065623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a2b80 0x7fc9141c6a40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00cc10 tx=0x7fc90c00cf20 comp rx=0 tx=0).stop 2024-09-16T07:18:38.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 1 -- 172.21.15.143:0/3481065623 shutdown_connections 2024-09-16T07:18:38.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 1 --2- 172.21.15.143:0/3481065623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9140a2b80 0x7fc9141c6a40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 1 --2- 172.21.15.143:0/3481065623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9140a1270 0x7fc9141c3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 1 --2- 172.21.15.143:0/3481065623 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9140a0e90 0x7fc9141c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 1 -- 172.21.15.143:0/3481065623 >> 172.21.15.143:0/3481065623 conn(0x7fc9140a4740 msgr2=0x7fc91409c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:38.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 1 -- 172.21.15.143:0/3481065623 shutdown_connections 2024-09-16T07:18:38.303 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-16T07:18:38.298+0000 7fc91b408700 1 -- 172.21.15.143:0/3481065623 wait complete. 2024-09-16T07:18:38.317 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T07:18:38.317 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T07:18:38.319 INFO:tasks.ceph.osd.0:Started 2024-09-16T07:18:38.319 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T07:18:38.319 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T07:18:38.321 INFO:tasks.ceph.osd.1:Started 2024-09-16T07:18:38.322 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T07:18:38.322 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T07:18:38.324 INFO:tasks.ceph.osd.2:Started 2024-09-16T07:18:38.324 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T07:18:38.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T07:18:38.326 INFO:tasks.ceph.osd.3:Started 2024-09-16T07:18:38.326 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T07:18:38.326 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T07:18:38.328 INFO:tasks.ceph.osd.4:Started 2024-09-16T07:18:38.328 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T07:18:38.328 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T07:18:38.330 INFO:tasks.ceph.osd.5:Started 2024-09-16T07:18:38.330 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T07:18:38.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.444+0000 7fe6bfdc8700 10 monclient: get_monmap_and_config 2024-09-16T07:18:38.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.444+0000 7fe6bfdc8700 10 monclient: build_initial_monmap 2024-09-16T07:18:38.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.444+0000 7fe6bfdc8700 10 monclient: monmap: 2024-09-16T07:18:38.447 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:38.447 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:38.447 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:38.451339+0000 2024-09-16T07:18:38.447 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:38.451339+0000 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 Processor -- start 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 -- start start 2024-09-16T07:18:38.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient: init 2024-09-16T07:18:38.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:38.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T07:18:38.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6b80a2850 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:38.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe6b80a0700 0x7fe6b8135d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:38.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient(hunting): picked mon.noname-c con 0x7fe6b80a0700 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b813aa60 0x7fe6b813ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient(hunting): picked mon.noname-b con 0x7fe6b813aa60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b806cec0 con 0x7fe6b80a2850 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b809fe20 con 0x7fe6b80a0700 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b806d100 con 0x7fe6b813aa60 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient(hunting): authenticate will time out at 838.937130s 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6be365700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b813aa60 0x7fe6b813ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bdb64700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe6b80a2850 0x7fe6b8135610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51834/0 (socket says 172.21.15.37:51834) 2024-09-16T07:18:38.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bdb64700 1 -- 172.21.15.37:0/3638078725 learned_addr learned my addr 172.21.15.37:0/3638078725 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6be365700 10 monclient(hunting): get_auth_request con 0x7fe6b813aa60 auth_method 0 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6be365700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 1 --2- 172.21.15.37:0/3638078725 >> v2:172.21.15.143:3300/0 conn(0x7fe6b80a0700 0x7fe6b8135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6be365700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6be365700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 10 monclient(hunting): get_auth_request con 0x7fe6b80a0700 auth_method 0 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6be365700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6be365700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6be365700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:38.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 1 -- 172.21.15.37:0/3638078725 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969705681 0 0) 0x7fe6b806cec0 con 0x7fe6b80a2850 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient(hunting): _negotiate 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient(hunting): my global_id is 4146 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 1 -- 172.21.15.37:0/3638078725 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a40035d0 con 0x7fe6b80a2850 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 10 monclient(hunting): handle_auth_done global_id 4168 payload 274 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 1 -- 172.21.15.37:0/3638078725 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b813aa60 msgr2=0x7fe6b813ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 1 --2- 172.21.15.37:0/3638078725 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b813aa60 0x7fe6b813ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 1 -- 172.21.15.37:0/3638078725 >> v1:172.21.15.37:6789/0 conn(0x7fe6b80a2850 legacy=0x7fe6b8135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 10 monclient: _finish_hunting 0 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 1 monclient: found mon.noname-c 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 1 -- 172.21.15.37:0/3638078725 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b809f4f0 con 0x7fe6b80a0700 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6be365700 1 --2- 172.21.15.37:0/3638078725 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b813aa60 0x7fe6b813ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bd363700 1 --2- 172.21.15.37:0/3638078725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b80a0700 0x7fe6b8135d20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009880 tx=0x7fe6ac009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccea51e723a84474 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 1 -- 172.21.15.37:0/3638078725 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac011830 con 0x7fe6b80a0700 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient: dump: 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:38.459 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient: _finish_auth 0 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient: _check_auth_tickets 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:38.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 5 monclient: authenticate success, global_id 4168 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 1 -- 172.21.15.37:0/3638078725 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ac011990 con 0x7fe6b80a0700 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 4 set_mon_vals no callback set 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 1 -- 172.21.15.37:0/3638078725 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac011d00 con 0x7fe6b80a0700 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bcb62700 10 monclient: dump: 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:38.461 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.462 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.462 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:38.462 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:38.462 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:38.462 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:38.463 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:38.463 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:38.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient: shutdown 2024-09-16T07:18:38.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:38.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/3638078725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b80a0700 msgr2=0x7fe6b8135d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:38.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 --2- 172.21.15.37:0/3638078725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b80a0700 0x7fe6b8135d20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009880 tx=0x7fe6ac009b90 comp rx=0 tx=0).stop 2024-09-16T07:18:38.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/3638078725 shutdown_connections 2024-09-16T07:18:38.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 --2- 172.21.15.37:0/3638078725 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b813aa60 0x7fe6b813ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 --2- 172.21.15.37:0/3638078725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b80a0700 0x7fe6b8135d20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/3638078725 >> 172.21.15.37:0/3638078725 conn(0x7fe6b80a4950 msgr2=0x7fe6b80a4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/3638078725 shutdown_connections 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/3638078725 wait complete. 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.448+0000 7fe6bfdc8700 10 monclient: build_initial_monmap 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient: monmap: 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 1 Processor -- start 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 1 -- start start 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient: init 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b80a0700 0x7fe6b81c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient(hunting): picked mon.noname-c con 0x7fe6b80a0700 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6b80a2850 0x7fe6b8133d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient(hunting): picked mon.noname-a con 0x7fe6b80a2850 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b813aa60 0x7fe6b81ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient(hunting): picked mon.noname-b con 0x7fe6b813aa60 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8120200 con 0x7fe6b80a2850 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b809e490 con 0x7fe6b813aa60 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b809f0b0 con 0x7fe6b80a0700 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient(hunting): authenticate will time out at 838.941058s 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6b80a2850 0x7fe6b8133d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:38.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b80a0700 0x7fe6b81c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6b80a2850 0x7fe6b8133d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47248/0 (socket says 172.21.15.37:47248) 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 1 -- 172.21.15.37:0/230332276 learned_addr learned my addr 172.21.15.37:0/230332276 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 10 monclient(hunting): get_auth_request con 0x7fe6b80a2850 auth_method 0 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 10 monclient(hunting): get_auth_request con 0x7fe6b80a0700 auth_method 0 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6be365700 1 --2- 172.21.15.37:0/230332276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b813aa60 0x7fe6b81ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6be365700 10 monclient(hunting): get_auth_request con 0x7fe6b813aa60 auth_method 0 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6be365700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6be365700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6be365700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6be365700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6be365700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6be365700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 10 monclient(hunting): handle_auth_done global_id 4154 payload 824 2024-09-16T07:18:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 1 -- 172.21.15.37:0/230332276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b813aa60 msgr2=0x7fe6b81ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 1 --2- 172.21.15.37:0/230332276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b813aa60 0x7fe6b81ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 1 -- 172.21.15.37:0/230332276 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6b80a2850 msgr2=0x7fe6b8133d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 1 --2- 172.21.15.37:0/230332276 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6b80a2850 0x7fe6b8133d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 10 monclient: _finish_hunting 0 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 1 monclient: found mon.noname-c 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 1 -- 172.21.15.37:0/230332276 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b40094b0 con 0x7fe6b80a0700 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bd363700 1 --2- 172.21.15.37:0/230332276 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6b80a2850 0x7fe6b8133d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bdb64700 1 --2- 172.21.15.37:0/230332276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b80a0700 0x7fe6b81c6ca0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000cbf0 tx=0x7fe6b000cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 1 -- 172.21.15.37:0/230332276 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b000a730 con 0x7fe6b80a0700 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 10 monclient: dump: 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 10 monclient: _finish_auth 0 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 1 -- 172.21.15.37:0/230332276 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b000a890 con 0x7fe6b80a0700 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 5 monclient: authenticate success, global_id 4154 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient: _renew_subs 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/230332276 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ac0094b0 con 0x7fe6b80a0700 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 1 -- 172.21.15.37:0/230332276 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b000ac00 con 0x7fe6b80a0700 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 10 monclient: dump: 2024-09-16T07:18:38.471 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6ab7fe700 4 set_mon_vals no callback set 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient: _renew_subs 2024-09-16T07:18:38.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:38.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/230332276 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b81203e0 con 0x7fe6b80a0700 2024-09-16T07:18:38.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 1 -- 172.21.15.37:0/230332276 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fe6b001c440 con 0x7fe6b80a0700 2024-09-16T07:18:38.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.452+0000 7fe6aa7fc700 1 -- 172.21.15.37:0/230332276 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fe6b000ae00 con 0x7fe6b80a0700 2024-09-16T07:18:38.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.456+0000 7fe6bfdc8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:38.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.456+0000 7fe6bfdc8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:38.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.456+0000 7fe6bfdc8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:38.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.456+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/230332276 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b80675e0 con 0x7fe6b80a0700 2024-09-16T07:18:38.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.456+0000 7fe6aa7fc700 1 -- 172.21.15.37:0/230332276 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fe6b80675e0 con 0x7fe6b80a0700 2024-09-16T07:18:38.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.456+0000 7fe6aa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:38.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.456+0000 7fe6aa7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:38.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.524+0000 7fe6bfdc8700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:38.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.524+0000 7fe6bfdc8700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:38.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.524+0000 7fe6bfdc8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:38.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.524+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/230332276 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6b811ec40 con 0x7fe6b80a0700 2024-09-16T07:18:38.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.524+0000 7fe6aa7fc700 1 -- 172.21.15.37:0/230332276 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (secure 0 0 0) 0x7fe6b0021060 con 0x7fe6b80a0700 2024-09-16T07:18:38.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.524+0000 7fe6aa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:38.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.524+0000 7fe6aa7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:38.526 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:38.527 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":8,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","created":"2024-09-16T07:18:36.646941+0000","modified":"2024-09-16T07:18:38.297482+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T07:18:38.297482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fd764fc8-b5fb-40fe-a64f-ce7b8c1df525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52d577a5-08b7-452d-aa35-e1560215a09e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3deb985-fb4e-42ee-8ed6-e6318cbca322","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ab447540-8ed1-4a0c-8f6c-dfeea8af02ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"be421fc0-1edf-4f20-b530-c9cf4eb20a99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8c7725bd-5f56-4fa0-b455-6569e313f9fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:18:38.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 10 monclient: shutdown 2024-09-16T07:18:38.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:38.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/230332276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b80a0700 msgr2=0x7fe6b81c6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:38.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 1 --2- 172.21.15.37:0/230332276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b80a0700 0x7fe6b81c6ca0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000cbf0 tx=0x7fe6b000cc20 comp rx=0 tx=0).stop 2024-09-16T07:18:38.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/230332276 shutdown_connections 2024-09-16T07:18:38.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 1 --2- 172.21.15.37:0/230332276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6b813aa60 0x7fe6b81ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 1 --2- 172.21.15.37:0/230332276 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6b80a2850 0x7fe6b8133d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 1 --2- 172.21.15.37:0/230332276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6b80a0700 0x7fe6b81c6ca0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:38.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/230332276 >> 172.21.15.37:0/230332276 conn(0x7fe6b80a4950 msgr2=0x7fe6b8134810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:38.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/230332276 shutdown_connections 2024-09-16T07:18:38.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:38.528+0000 7fe6bfdc8700 1 -- 172.21.15.37:0/230332276 wait complete. 2024-09-16T07:18:38.536 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:38.534+0000 7f8b087a32c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T07:18:38.539 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T07:18:38.540 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T07:18:38.586 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:38.582+0000 7f8b087a32c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T07:18:38.693 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T07:18:38.692+0000 7fce22d9e880 -1 Falling back to public interface 2024-09-16T07:18:38.704 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-16T07:18:38.704+0000 7fd44ca31880 -1 Falling back to public interface 2024-09-16T07:18:38.708 INFO:tasks.ceph.osd.5.smithi143.stderr:2024-09-16T07:18:38.706+0000 7fa7f7e2a880 -1 Falling back to public interface 2024-09-16T07:18:38.756 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:38.754+0000 7f8b087a32c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T07:18:38.820 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:38.814+0000 7f8b087a32c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T07:18:38.881 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:38.878+0000 7f8b087a32c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T07:18:38.937 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-09-16T07:18:38.934+0000 7f5e31a6f880 -1 Falling back to public interface 2024-09-16T07:18:38.937 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T07:18:38.934+0000 7f444e615880 -1 Falling back to public interface 2024-09-16T07:18:38.948 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-09-16T07:18:38.948+0000 7f58f0403880 -1 Falling back to public interface 2024-09-16T07:18:39.143 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T07:18:39.253 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.252+0000 7fbc9b011700 10 monclient: get_monmap_and_config 2024-09-16T07:18:39.253 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.252+0000 7fbc9b011700 10 monclient: build_initial_monmap 2024-09-16T07:18:39.253 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.252+0000 7fbc9b011700 10 monclient: monmap: 2024-09-16T07:18:39.253 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:39.257928+0000 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:39.257928+0000 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:39.254 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:39.255 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 1 Processor -- start 2024-09-16T07:18:39.255 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 1 -- start start 2024-09-16T07:18:39.255 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient: init 2024-09-16T07:18:39.258 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:39.258 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T07:18:39.258 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbc940a4d00 0x7fbc9406a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:39.258 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient(hunting): picked mon.noname-c con 0x7fbc940a4d00 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc94099fc0 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94136be0 0x7fbc94138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient(hunting): picked mon.noname-b con 0x7fbc94136be0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc940a50e0 con 0x7fbc94099fc0 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9409b810 con 0x7fbc940a4d00 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc94099d80 con 0x7fbc94136be0 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9980e700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fbc94099fc0 0x7fbc9406ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51850/0 (socket says 172.21.15.37:51850) 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9980e700 1 -- 172.21.15.37:0/3470908850 learned_addr learned my addr 172.21.15.37:0/3470908850 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 1 --2- 172.21.15.37:0/3470908850 >> v2:172.21.15.143:3300/0 conn(0x7fbc940a4d00 0x7fbc9406a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient(hunting): authenticate will time out at 839.746132s 2024-09-16T07:18:39.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 10 monclient(hunting): get_auth_request con 0x7fbc940a4d00 auth_method 0 2024-09-16T07:18:39.260 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:39.260 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:39.260 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:39.260 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a810700 1 --2- 172.21.15.37:0/3470908850 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94136be0 0x7fbc94138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:39.260 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a810700 10 monclient(hunting): get_auth_request con 0x7fbc94136be0 auth_method 0 2024-09-16T07:18:39.260 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a810700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:39.260 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a810700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:39.260 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a810700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:39.260 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a810700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a810700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a810700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 1 -- 172.21.15.37:0/3470908850 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701747299 0 0) 0x7fbc940a50e0 con 0x7fbc94099fc0 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient(hunting): _negotiate 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient(hunting): my global_id is 4152 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 1 -- 172.21.15.37:0/3470908850 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc800057e0 con 0x7fbc94099fc0 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 10 monclient(hunting): handle_auth_done global_id 4174 payload 274 2024-09-16T07:18:39.261 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 1 -- 172.21.15.37:0/3470908850 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94136be0 msgr2=0x7fbc94138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 1 --2- 172.21.15.37:0/3470908850 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94136be0 0x7fbc94138fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 1 -- 172.21.15.37:0/3470908850 >> v1:172.21.15.37:6789/0 conn(0x7fbc94099fc0 legacy=0x7fbc9406ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 10 monclient: _finish_hunting 0 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 1 monclient: found mon.noname-c 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 1 -- 172.21.15.37:0/3470908850 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9409cbf0 con 0x7fbc940a4d00 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9a00f700 1 --2- 172.21.15.37:0/3470908850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc940a4d00 0x7fbc9406a850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009850 tx=0x7fbc90009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b66141aa7ce0bc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 1 -- 172.21.15.37:0/3470908850 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90011830 con 0x7fbc940a4d00 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient: dump: 2024-09-16T07:18:39.262 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient: _finish_auth 0 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient: _check_auth_tickets 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 1 -- 172.21.15.37:0/3470908850 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc90011990 con 0x7fbc940a4d00 2024-09-16T07:18:39.263 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 5 monclient: authenticate success, global_id 4174 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 1 -- 172.21.15.37:0/3470908850 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90011d40 con 0x7fbc940a4d00 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9b011700 4 set_mon_vals no callback set 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.256+0000 7fbc9900d700 10 monclient: dump: 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:39.264 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:39.265 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:39.265 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:39.265 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:39.265 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:39.265 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:39.269 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 10 monclient: shutdown 2024-09-16T07:18:39.269 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:39.269 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 1 -- 172.21.15.37:0/3470908850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc940a4d00 msgr2=0x7fbc9406a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:39.269 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 1 --2- 172.21.15.37:0/3470908850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc940a4d00 0x7fbc9406a850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009850 tx=0x7fbc90009b60 comp rx=0 tx=0).stop 2024-09-16T07:18:39.269 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 1 -- 172.21.15.37:0/3470908850 shutdown_connections 2024-09-16T07:18:39.269 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 1 --2- 172.21.15.37:0/3470908850 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94136be0 0x7fbc94138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 1 --2- 172.21.15.37:0/3470908850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc940a4d00 0x7fbc9406a850 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 1 -- 172.21.15.37:0/3470908850 >> 172.21.15.37:0/3470908850 conn(0x7fbc940a41e0 msgr2=0x7fbc940a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 1 -- 172.21.15.37:0/3470908850 shutdown_connections 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 1 -- 172.21.15.37:0/3470908850 wait complete. 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 10 monclient: build_initial_monmap 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 10 monclient: monmap: 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:39.270 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 1 Processor -- start 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 1 -- start start 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.260+0000 7fbc9b011700 10 monclient: init 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94099fc0 0x7fbc94131520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient(hunting): picked mon.noname-c con 0x7fbc94099fc0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc940a4d00 0x7fbc94131a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient(hunting): picked mon.noname-b con 0x7fbc940a4d00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc94136be0 0x7fbc9411a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient(hunting): picked mon.noname-a con 0x7fbc94136be0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9409b460 con 0x7fbc94136be0 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9411d640 con 0x7fbc940a4d00 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9409f300 con 0x7fbc94099fc0 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94099fc0 0x7fbc94131520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient(hunting): authenticate will time out at 839.753290s 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94099fc0 0x7fbc94131520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:55024/0 (socket says 172.21.15.37:55024) 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 1 -- 172.21.15.37:0/2867680402 learned_addr learned my addr 172.21.15.37:0/2867680402 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 10 monclient(hunting): get_auth_request con 0x7fbc94099fc0 auth_method 0 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a810700 1 --2- 172.21.15.37:0/2867680402 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc94136be0 0x7fbc9411a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9980e700 1 --2- 172.21.15.37:0/2867680402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc940a4d00 0x7fbc94131a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a810700 10 monclient(hunting): get_auth_request con 0x7fbc94136be0 auth_method 0 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a810700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a810700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a810700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9980e700 10 monclient(hunting): get_auth_request con 0x7fbc940a4d00 auth_method 0 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9980e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9980e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9980e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:39.273 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9980e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9980e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9980e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a810700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a810700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a810700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 10 monclient(hunting): handle_auth_done global_id 4160 payload 824 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 1 -- 172.21.15.37:0/2867680402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc940a4d00 msgr2=0x7fbc94131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 1 --2- 172.21.15.37:0/2867680402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc940a4d00 0x7fbc94131a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 1 -- 172.21.15.37:0/2867680402 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc94136be0 msgr2=0x7fbc9411a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 1 --2- 172.21.15.37:0/2867680402 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc94136be0 0x7fbc9411a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 10 monclient: _finish_hunting 0 2024-09-16T07:18:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 1 monclient: found mon.noname-c 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 1 -- 172.21.15.37:0/2867680402 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc900094b0 con 0x7fbc94099fc0 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9a00f700 1 --2- 172.21.15.37:0/2867680402 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94099fc0 0x7fbc94131520 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009eb0 tx=0x7fbc90004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 1 -- 172.21.15.37:0/2867680402 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90004510 con 0x7fbc94099fc0 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 10 monclient: dump: 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:39.275 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 10 monclient: _finish_auth 0 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 5 monclient: authenticate success, global_id 4160 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient: _renew_subs 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 1 -- 172.21.15.37:0/2867680402 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc94135eb0 con 0x7fbc94099fc0 2024-09-16T07:18:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 1 -- 172.21.15.37:0/2867680402 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc900036c0 con 0x7fbc94099fc0 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient: _renew_subs 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9b011700 1 -- 172.21.15.37:0/2867680402 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9411def0 con 0x7fbc94099fc0 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc9880c700 4 set_mon_vals no callback set 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 1 -- 172.21.15.37:0/2867680402 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90003a30 con 0x7fbc94099fc0 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 10 monclient: dump: 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:39.277 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 1 -- 172.21.15.37:0/2867680402 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fbc90021070 con 0x7fbc94099fc0 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.264+0000 7fbc8affd700 1 -- 172.21.15.37:0/2867680402 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fbc90024ba0 con 0x7fbc94099fc0 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.268+0000 7fbc9b011700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.268+0000 7fbc9b011700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.268+0000 7fbc9b011700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:39.278 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.268+0000 7fbc9b011700 1 -- 172.21.15.37:0/2867680402 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc940738e0 con 0x7fbc94099fc0 2024-09-16T07:18:39.279 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.268+0000 7fbc8affd700 1 -- 172.21.15.37:0/2867680402 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fbc90044e30 con 0x7fbc94099fc0 2024-09-16T07:18:39.279 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.268+0000 7fbc8affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:39.279 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.268+0000 7fbc8affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:39.284 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:39.282+0000 7f8b087a32c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T07:18:39.335 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:39.336 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:39.336 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:39.336 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 1 -- 172.21.15.37:0/2867680402 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc9411c710 con 0x7fbc94099fc0 2024-09-16T07:18:39.336 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc8affd700 1 -- 172.21.15.37:0/2867680402 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5604 (secure 0 0 0) 0x7fbc90033030 con 0x7fbc94099fc0 2024-09-16T07:18:39.337 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc8affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:39.337 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc8affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:39.337 INFO:teuthology.misc.health.smithi037.stdout: 2024-09-16T07:18:39.337 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":8,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","created":"2024-09-16T07:18:36.646941+0000","modified":"2024-09-16T07:18:38.297482+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T07:18:38.297482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fd764fc8-b5fb-40fe-a64f-ce7b8c1df525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52d577a5-08b7-452d-aa35-e1560215a09e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e3deb985-fb4e-42ee-8ed6-e6318cbca322","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ab447540-8ed1-4a0c-8f6c-dfeea8af02ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"be421fc0-1edf-4f20-b530-c9cf4eb20a99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8c7725bd-5f56-4fa0-b455-6569e313f9fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:18:39.338 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 10 monclient: shutdown 2024-09-16T07:18:39.338 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:39.338 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 1 -- 172.21.15.37:0/2867680402 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94099fc0 msgr2=0x7fbc94131520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:39.338 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 1 --2- 172.21.15.37:0/2867680402 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94099fc0 0x7fbc94131520 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009eb0 tx=0x7fbc90004770 comp rx=0 tx=0).stop 2024-09-16T07:18:39.338 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 1 -- 172.21.15.37:0/2867680402 shutdown_connections 2024-09-16T07:18:39.338 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 1 --2- 172.21.15.37:0/2867680402 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc94136be0 0x7fbc9411a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 1 --2- 172.21.15.37:0/2867680402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc940a4d00 0x7fbc94131a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 1 --2- 172.21.15.37:0/2867680402 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc94099fc0 0x7fbc94131520 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 1 -- 172.21.15.37:0/2867680402 >> 172.21.15.37:0/2867680402 conn(0x7fbc940a41e0 msgr2=0x7fbc9409ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 1 -- 172.21.15.37:0/2867680402 shutdown_connections 2024-09-16T07:18:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:39.336+0000 7fbc9b011700 1 -- 172.21.15.37:0/2867680402 wait complete. 2024-09-16T07:18:39.349 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T07:18:39.400 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:39.398+0000 7f8b087a32c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T07:18:39.735 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:39.734+0000 7f8b087a32c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T07:18:39.921 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:39.918+0000 7f8b087a32c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T07:18:39.989 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:39.986+0000 7f8b087a32c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T07:18:40.151 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:40.150+0000 7f8b087a32c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T07:18:40.915 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:40.910+0000 7f8b087a32c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T07:18:40.985 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:40.982+0000 7f8b087a32c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T07:18:41.150 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:41.146+0000 7f8b087a32c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T07:18:41.217 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:41.214+0000 7f8b087a32c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T07:18:41.290 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:41.286+0000 7f8b087a32c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T07:18:41.620 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:41.618+0000 7f8b087a32c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T07:18:41.701 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:41.698+0000 7f8b087a32c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T07:18:42.275 INFO:tasks.ceph.osd.5.smithi143.stderr:2024-09-16T07:18:42.274+0000 7fa7f7e2a880 -1 osd.5 0 log_to_monitors true 2024-09-16T07:18:42.405 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:42.402+0000 7f8b087a32c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T07:18:42.463 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T07:18:42.464+0000 7fce22d9e880 -1 osd.0 0 log_to_monitors true 2024-09-16T07:18:42.518 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-09-16T07:18:42.516+0000 7f58f0403880 -1 osd.2 0 log_to_monitors true 2024-09-16T07:18:42.530 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-16T07:18:42.528+0000 7fd44ca31880 -1 osd.1 0 log_to_monitors true 2024-09-16T07:18:42.790 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-09-16T07:18:42.786+0000 7f5e31a6f880 -1 osd.3 0 log_to_monitors true 2024-09-16T07:18:42.796 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T07:18:42.794+0000 7f444e615880 -1 osd.4 0 log_to_monitors true 2024-09-16T07:18:43.358 INFO:tasks.ceph.osd.5.smithi143.stderr:2024-09-16T07:18:43.354+0000 7fa7f4df1700 -1 osd.5 0 waiting for initial osdmap 2024-09-16T07:18:43.368 INFO:tasks.ceph.osd.5.smithi143.stderr:2024-09-16T07:18:43.366+0000 7fa7ef723700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T07:18:43.448 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:43.446+0000 7f8aef318700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T07:18:43.449 INFO:tasks.ceph.mgr.x.smithi143.stderr:2024-09-16T07:18:43.446+0000 7f8aef318700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T07:18:44.360 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-16T07:18:44.360+0000 7fd44804d700 -1 osd.1 0 waiting for initial osdmap 2024-09-16T07:18:44.382 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T07:18:44.378+0000 7f4449c31700 -1 osd.4 0 waiting for initial osdmap 2024-09-16T07:18:44.389 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-16T07:18:44.388+0000 7fd44432a700 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T07:18:44.396 INFO:tasks.ceph.osd.4.smithi143.stderr:2024-09-16T07:18:44.394+0000 7f4445f0e700 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T07:18:45.355 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-09-16T07:18:45.354+0000 7f5e2ea36700 -1 osd.3 0 waiting for initial osdmap 2024-09-16T07:18:45.359 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-09-16T07:18:45.358+0000 7f5e25360700 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T07:18:45.788 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-09-16T07:18:45.788+0000 7f58ed3ca700 -1 osd.2 0 waiting for initial osdmap 2024-09-16T07:18:45.793 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-09-16T07:18:45.792+0000 7f58e3cf4700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T07:18:45.813 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T07:18:45.812+0000 7fce1e3ba700 -1 osd.0 0 waiting for initial osdmap 2024-09-16T07:18:45.821 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-09-16T07:18:45.820+0000 7fce1668f700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T07:18:45.955 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T07:18:46.063 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient: get_monmap_and_config 2024-09-16T07:18:46.063 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient: build_initial_monmap 2024-09-16T07:18:46.063 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient: monmap: 2024-09-16T07:18:46.063 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-09-16T07:18:46.063 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:46.063 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:46.067634+0000 2024-09-16T07:18:46.063 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:46.067634+0000 2024-09-16T07:18:46.064 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:46.064 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:46.064 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:46.064 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:46.064 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:46.064 INFO:teuthology.misc.health.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:46.064 INFO:teuthology.misc.health.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:46.064 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 1 Processor -- start 2024-09-16T07:18:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 1 -- start start 2024-09-16T07:18:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient: init 2024-09-16T07:18:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:46.069 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T07:18:46.070 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facc40a13e0 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:46.070 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a1ed0 0x7facc4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:46.070 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient(hunting): picked mon.noname-b con 0x7facc40a1ed0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:46.070 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7facc41385e0 0x7facc413a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:46.070 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient(hunting): picked mon.noname-a con 0x7facc41385e0 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:46.070 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc4070350 con 0x7facc40a13e0 2024-09-16T07:18:46.070 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:46.070 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc40a22b0 con 0x7facc41385e0 2024-09-16T07:18:46.070 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc4071bd0 con 0x7facc40a1ed0 2024-09-16T07:18:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient(hunting): authenticate will time out at 846.553588s 2024-09-16T07:18:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7facc41385e0 0x7facc413a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc37fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a1ed0 0x7facc4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7facc41385e0 0x7facc413a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56464/0 (socket says 172.21.15.37:56464) 2024-09-16T07:18:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc37fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a1ed0 0x7facc4071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44856/0 (socket says 172.21.15.37:44856) 2024-09-16T07:18:46.071 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 1 -- 172.21.15.37:0/563226262 learned_addr learned my addr 172.21.15.37:0/563226262 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 10 monclient(hunting): get_auth_request con 0x7facc41385e0 auth_method 0 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc37fe700 10 monclient(hunting): get_auth_request con 0x7facc40a1ed0 auth_method 0 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:46.072 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 10 monclient(hunting): handle_auth_done global_id 4191 payload 274 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 1 -- 172.21.15.37:0/563226262 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a1ed0 msgr2=0x7facc4071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 1 --2- 172.21.15.37:0/563226262 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a1ed0 0x7facc4071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 1 -- 172.21.15.37:0/563226262 >> v1:172.21.15.143:6789/0 conn(0x7facc40a13e0 legacy=0x7facc40a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 10 monclient: _finish_hunting 0 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 1 monclient: found mon.noname-a 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 1 -- 172.21.15.37:0/563226262 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc406ffe0 con 0x7facc41385e0 2024-09-16T07:18:46.073 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc886e700 1 --2- 172.21.15.37:0/563226262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 0x7facc413a9d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7facbc009910 tx=0x7facbc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1100fe95ec90ac74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 1 -- 172.21.15.37:0/563226262 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc011830 con 0x7facc41385e0 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 10 monclient: dump: 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:46.074 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 10 monclient: _finish_auth 0 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 1 -- 172.21.15.37:0/563226262 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc011990 con 0x7facc41385e0 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 5 monclient: authenticate success, global_id 4191 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 4 set_mon_vals no callback set 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 1 -- 172.21.15.37:0/563226262 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc011d00 con 0x7facc41385e0 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc2ffd700 10 monclient: dump: 2024-09-16T07:18:46.075 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 10 monclient: shutdown 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 1 -- 172.21.15.37:0/563226262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 msgr2=0x7facc413a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:46.076 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.064+0000 7facc906f700 1 --2- 172.21.15.37:0/563226262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 0x7facc413a9d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7facbc009910 tx=0x7facbc009c20 comp rx=0 tx=0).stop 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 -- 172.21.15.37:0/563226262 shutdown_connections 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 --2- 172.21.15.37:0/563226262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 0x7facc413a9d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 --2- 172.21.15.37:0/563226262 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a1ed0 0x7facc4071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 -- 172.21.15.37:0/563226262 >> 172.21.15.37:0/563226262 conn(0x7facc40a6000 msgr2=0x7facc40a24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 -- 172.21.15.37:0/563226262 shutdown_connections 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 -- 172.21.15.37:0/563226262 wait complete. 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient: build_initial_monmap 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient: monmap: 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:46.077 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:46.078 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:46.078 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:46.078 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:46.078 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:46.078 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 Processor -- start 2024-09-16T07:18:46.078 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 -- start start 2024-09-16T07:18:46.078 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient: init 2024-09-16T07:18:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:18:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a13e0 0x7facc4133220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient(hunting): picked mon.noname-c con 0x7facc40a13e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facc40a1ed0 0x7facc4133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:46.082 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient(hunting): picked mon.noname-b con 0x7facc40a1ed0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 0x7facc409dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient(hunting): picked mon.noname-a con 0x7facc41385e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc411da30 con 0x7facc41385e0 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc4072660 con 0x7facc40a1ed0 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc409f6e0 con 0x7facc40a13e0 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc906f700 10 monclient(hunting): authenticate will time out at 846.559441s 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc886e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 0x7facc409dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc886e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 0x7facc409dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56480/0 (socket says 172.21.15.37:56480) 2024-09-16T07:18:46.083 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc886e700 1 -- 172.21.15.37:0/3984775437 learned_addr learned my addr 172.21.15.37:0/3984775437 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc886e700 10 monclient(hunting): get_auth_request con 0x7facc41385e0 auth_method 0 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc886e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc886e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc886e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc3fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a13e0 0x7facc4133220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc886e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc886e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc886e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc37fe700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facc40a1ed0 0x7facc4133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc3fff700 10 monclient(hunting): get_auth_request con 0x7facc40a13e0 auth_method 0 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:46.084 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc37fe700 10 monclient(hunting): get_auth_request con 0x7facc40a1ed0 auth_method 0 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.068+0000 7facc37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 10 monclient(hunting): handle_auth_done global_id 4194 payload 824 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 1 -- 172.21.15.37:0/3984775437 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a13e0 msgr2=0x7facc4133220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a13e0 0x7facc4133220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 1 -- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facc40a1ed0 msgr2=0x7facc4133760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facc40a1ed0 0x7facc4133760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 10 monclient: _finish_hunting 0 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 1 monclient: found mon.noname-a 2024-09-16T07:18:46.085 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 1 -- 172.21.15.37:0/3984775437 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc009570 con 0x7facc41385e0 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc37fe700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facc40a1ed0 0x7facc4133760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc3fff700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a13e0 0x7facc4133220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc886e700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 0x7facc409dae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7facc40a31f0 tx=0x7facbc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 1 -- 172.21.15.37:0/3984775437 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc00b930 con 0x7facc41385e0 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 10 monclient: dump: 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:46.086 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 10 monclient: _finish_auth 0 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 5 monclient: authenticate success, global_id 4194 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 10 monclient: _renew_subs 2024-09-16T07:18:46.087 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 1 -- 172.21.15.37:0/3984775437 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc00bab0 con 0x7facc41385e0 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 1 -- 172.21.15.37:0/3984775437 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc411d470 con 0x7facc41385e0 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 1 -- 172.21.15.37:0/3984775437 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc00be60 con 0x7facc41385e0 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc27fc700 4 set_mon_vals no callback set 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 10 monclient: dump: 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:46.088 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 10 monclient: _renew_subs 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 1 -- 172.21.15.37:0/3984775437 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc411c020 con 0x7facc41385e0 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 1 -- 172.21.15.37:0/3984775437 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) v1 ==== 43251+0+0 (secure 0 0 0) 0x7facbc021070 con 0x7facc41385e0 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faca4035570 0x7faca4037a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc0ff9700 1 -- 172.21.15.37:0/3984775437 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2767+0+0 (secure 0 0 0) 0x7facbc048900 con 0x7facc41385e0 2024-09-16T07:18:46.089 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:46.090 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:46.090 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc906f700 1 -- 172.21.15.37:0/3984775437 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc4071260 con 0x7facc41385e0 2024-09-16T07:18:46.090 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc3fff700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faca4035570 0x7faca4037a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:46.090 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc3fff700 10 monclient: get_auth_request con 0x7faca4035570 auth_method 0 2024-09-16T07:18:46.090 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.072+0000 7facc3fff700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faca4035570 0x7faca4037a30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7facb40017a0 tx=0x7facb4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:46.090 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.080+0000 7facc0ff9700 1 -- 172.21.15.37:0/3984775437 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7facbc015020 con 0x7facc41385e0 2024-09-16T07:18:46.090 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.080+0000 7facc0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:46.090 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.080+0000 7facc0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:46.254 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.252+0000 7facc906f700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:46.254 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.252+0000 7facc906f700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:46.254 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.252+0000 7facc906f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:46.254 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.252+0000 7facc906f700 1 -- 172.21.15.37:0/3984775437 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facc4069670 con 0x7facc41385e0 2024-09-16T07:18:46.255 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc0ff9700 1 -- 172.21.15.37:0/3984775437 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7062 (secure 0 0 0) 0x7facbc032070 con 0x7facc41385e0 2024-09-16T07:18:46.255 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:46.255 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:46.255 INFO:teuthology.misc.health.smithi037.stdout: 2024-09-16T07:18:46.256 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":12,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","created":"2024-09-16T07:18:36.646941+0000","modified":"2024-09-16T07:18:45.367803+0000","last_up_change":"2024-09-16T07:18:45.367803+0000","last_in_change":"2024-09-16T07:18:38.297482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fd764fc8-b5fb-40fe-a64f-ce7b8c1df525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52d577a5-08b7-452d-aa35-e1560215a09e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6809","nonce":810396637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6811","nonce":810396637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6815","nonce":810396637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6813","nonce":810396637}]},"public_addr":"172.21.15.37:6809/810396637","cluster_addr":"172.21.15.37:6811/810396637","heartbeat_back_addr":"172.21.15.37:6815/810396637","heartbeat_front_addr":"172.21.15.37:6813/810396637","state":["exists","up"]},{"osd":2,"uuid":"e3deb985-fb4e-42ee-8ed6-e6318cbca322","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ab447540-8ed1-4a0c-8f6c-dfeea8af02ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"be421fc0-1edf-4f20-b530-c9cf4eb20a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6810","nonce":3355421705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6814","nonce":3355421705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6822","nonce":3355421705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6818","nonce":3355421705}]},"public_addr":"172.21.15.143:6810/3355421705","cluster_addr":"172.21.15.143:6814/3355421705","heartbeat_back_addr":"172.21.15.143:6822/3355421705","heartbeat_front_addr":"172.21.15.143:6818/3355421705","state":["exists","up"]},{"osd":5,"uuid":"8c7725bd-5f56-4fa0-b455-6569e313f9fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6801","nonce":3921632375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6803","nonce":3921632375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6807","nonce":3921632375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6805","nonce":3921632375}]},"public_addr":"172.21.15.143:6801/3921632375","cluster_addr":"172.21.15.143:6803/3921632375","heartbeat_back_addr":"172.21.15.143:6807/3921632375","heartbeat_front_addr":"172.21.15.143:6805/3921632375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:18:46.258 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 -- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faca4035570 msgr2=0x7faca4037a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faca4035570 0x7faca4037a30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7facb40017a0 tx=0x7facb4009040 comp rx=0 tx=0).stop 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 10 monclient: shutdown 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 -- 172.21.15.37:0/3984775437 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 msgr2=0x7facc409dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 0x7facc409dae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7facc40a31f0 tx=0x7facbc004770 comp rx=0 tx=0).stop 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 -- 172.21.15.37:0/3984775437 shutdown_connections 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc41385e0 0x7facc409dae0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7facc40a1ed0 0x7facc4133760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faca4035570 0x7faca4037a30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 --2- 172.21.15.37:0/3984775437 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc40a13e0 0x7facc4133220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 -- 172.21.15.37:0/3984775437 >> 172.21.15.37:0/3984775437 conn(0x7facc40a6000 msgr2=0x7facc40a38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 -- 172.21.15.37:0/3984775437 shutdown_connections 2024-09-16T07:18:46.259 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:46.256+0000 7facc906f700 1 -- 172.21.15.37:0/3984775437 wait complete. 2024-09-16T07:18:46.269 DEBUG:teuthology.misc:3 of 6 OSDs are up 2024-09-16T07:18:52.875 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T07:18:52.985 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient: get_monmap_and_config 2024-09-16T07:18:52.985 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient: build_initial_monmap 2024-09-16T07:18:52.985 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient: monmap: 2024-09-16T07:18:52.985 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-09-16T07:18:52.985 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:52.985 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:52.989171+0000 2024-09-16T07:18:52.985 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:52.989171+0000 2024-09-16T07:18:52.986 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:52.986 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:52.986 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:52.986 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:52.986 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:52.986 INFO:teuthology.misc.health.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:52.986 INFO:teuthology.misc.health.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:52.986 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:52.997 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 1 Processor -- start 2024-09-16T07:18:52.997 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 1 -- start start 2024-09-16T07:18:52.997 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient: init 2024-09-16T07:18:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T07:18:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7c7409f2c0 0x7f7c7409d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient(hunting): picked mon.noname-c con 0x7f7c7409f2c0 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c7409af30 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c7409fdb0 0x7f7c741394e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient(hunting): picked mon.noname-a con 0x7f7c7409fdb0 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7409b3b0 con 0x7f7c7409af30 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7409ad90 con 0x7f7c7409fdb0 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7409caa0 con 0x7f7c7409f2c0 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7a09e700 10 monclient(hunting): authenticate will time out at 853.475289s 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7989d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c7409fdb0 0x7f7c741394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7889b700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f7c7409af30 0x7f7c7409f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36570/0 (socket says 172.21.15.37:36570) 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7889b700 1 -- 172.21.15.37:0/1309161657 learned_addr learned my addr 172.21.15.37:0/1309161657 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7989d700 10 monclient(hunting): get_auth_request con 0x7f7c7409fdb0 auth_method 0 2024-09-16T07:18:52.999 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7989d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:53.000 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7989d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:53.000 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7989d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:53.000 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7989d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:53.000 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7989d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:53.000 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7909c700 1 --2- 172.21.15.37:0/1309161657 >> v2:172.21.15.143:3300/0 conn(0x7f7c7409f2c0 0x7f7c7409d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.000 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7989d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7909c700 10 monclient(hunting): get_auth_request con 0x7f7c7409f2c0 auth_method 0 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7909c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7909c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.984+0000 7f7c7909c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7909c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7909c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7909c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7989d700 10 monclient(hunting): handle_auth_done global_id 4206 payload 274 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7989d700 1 -- 172.21.15.37:0/1309161657 >> v2:172.21.15.143:3300/0 conn(0x7f7c7409f2c0 msgr2=0x7f7c7409d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7989d700 1 --2- 172.21.15.37:0/1309161657 >> v2:172.21.15.143:3300/0 conn(0x7f7c7409f2c0 0x7f7c7409d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7989d700 1 -- 172.21.15.37:0/1309161657 >> v1:172.21.15.37:6789/0 conn(0x7f7c7409af30 legacy=0x7f7c7409f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:53.001 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7989d700 10 monclient: _finish_hunting 0 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7989d700 1 monclient: found mon.noname-a 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7989d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7989d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7989d700 1 -- 172.21.15.37:0/1309161657 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7409dc60 con 0x7f7c7409fdb0 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 1 -- 172.21.15.37:0/1309161657 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999866189 0 0) 0x7f7c7409b3b0 con 0x7f7c7409af30 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7989d700 1 --2- 172.21.15.37:0/1309161657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409fdb0 0x7f7c741394e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7c7406b890 tx=0x7f7c6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ecc579337eac3a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 1 -- 172.21.15.37:0/1309161657 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00e070 con 0x7f7c7409fdb0 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:53.002 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 10 monclient: dump: 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 10 monclient: _finish_auth 0 2024-09-16T07:18:53.003 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 10 monclient: _check_auth_tickets 2024-09-16T07:18:53.004 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:53.004 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 1 -- 172.21.15.37:0/1309161657 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c00bae0 con 0x7f7c7409fdb0 2024-09-16T07:18:53.004 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 5 monclient: authenticate success, global_id 4206 2024-09-16T07:18:53.004 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:53.004 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:53.004 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:53.004 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 1 -- 172.21.15.37:0/1309161657 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00be90 con 0x7f7c7409fdb0 2024-09-16T07:18:53.004 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:53.004 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 4 set_mon_vals no callback set 2024-09-16T07:18:53.004 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c6bfff700 10 monclient: dump: 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:53.005 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:53.006 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 10 monclient: shutdown 2024-09-16T07:18:53.006 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:53.006 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 1 -- 172.21.15.37:0/1309161657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409fdb0 msgr2=0x7f7c741394e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:53.006 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 1 --2- 172.21.15.37:0/1309161657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409fdb0 0x7f7c741394e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7c7406b890 tx=0x7f7c6c00b330 comp rx=0 tx=0).stop 2024-09-16T07:18:53.006 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 1 -- 172.21.15.37:0/1309161657 shutdown_connections 2024-09-16T07:18:53.006 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 1 --2- 172.21.15.37:0/1309161657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409fdb0 0x7f7c741394e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.006 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 1 --2- 172.21.15.37:0/1309161657 >> v2:172.21.15.143:3300/0 conn(0x7f7c7409f2c0 0x7f7c7409d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.006 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 1 -- 172.21.15.37:0/1309161657 >> 172.21.15.37:0/1309161657 conn(0x7f7c740a4060 msgr2=0x7f7c740a03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:53.006 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 1 -- 172.21.15.37:0/1309161657 shutdown_connections 2024-09-16T07:18:53.006 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 1 -- 172.21.15.37:0/1309161657 wait complete. 2024-09-16T07:18:53.007 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 10 monclient: build_initial_monmap 2024-09-16T07:18:53.007 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.988+0000 7f7c7a09e700 10 monclient: monmap: 2024-09-16T07:18:53.007 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-09-16T07:18:53.007 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:53.007 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:53.007 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-09-16T07:18:53.007 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:53.007 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:53.007 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 1 Processor -- start 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 1 -- start start 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient: init 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409af30 0x7f7c741ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient(hunting): picked mon.noname-a con 0x7f7c7409af30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:53.008 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c7409f2c0 0x7f7c741caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient(hunting): picked mon.noname-c con 0x7f7c7409f2c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c7409fdb0 0x7f7c74120940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient(hunting): picked mon.noname-b con 0x7f7c7409fdb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c74123150 con 0x7f7c7409af30 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c741245e0 con 0x7f7c7409fdb0 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c74131190 con 0x7f7c7409f2c0 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient(hunting): authenticate will time out at 853.482304s 2024-09-16T07:18:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409af30 0x7f7c741ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409af30 0x7f7c741ca550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36322/0 (socket says 172.21.15.37:36322) 2024-09-16T07:18:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 1 -- 172.21.15.37:0/138285213 learned_addr learned my addr 172.21.15.37:0/138285213 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 10 monclient(hunting): get_auth_request con 0x7f7c7409af30 auth_method 0 2024-09-16T07:18:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7889b700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c7409f2c0 0x7f7c741caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7889b700 10 monclient(hunting): get_auth_request con 0x7f7c7409f2c0 auth_method 0 2024-09-16T07:18:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7889b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7889b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7889b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7989d700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c7409fdb0 0x7f7c74120940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7989d700 10 monclient(hunting): get_auth_request con 0x7f7c7409fdb0 auth_method 0 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7989d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7989d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7989d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7889b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7889b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7889b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7989d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7989d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7989d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 10 monclient(hunting): handle_auth_done global_id 4212 payload 824 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 1 -- 172.21.15.37:0/138285213 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c7409f2c0 msgr2=0x7f7c741caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c7409f2c0 0x7f7c741caa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 1 -- 172.21.15.37:0/138285213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c7409fdb0 msgr2=0x7f7c74120940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c7409fdb0 0x7f7c74120940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 10 monclient: _finish_hunting 0 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 1 monclient: found mon.noname-a 2024-09-16T07:18:53.012 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 1 -- 172.21.15.37:0/138285213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6c009a70 con 0x7f7c7409af30 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7909c700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409af30 0x7f7c741ca550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400e9d0 tx=0x7f7c6400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 1 -- 172.21.15.37:0/138285213 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64009720 con 0x7f7c7409af30 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 10 monclient: dump: 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.013 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 10 monclient: _finish_auth 0 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 1 -- 172.21.15.37:0/138285213 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6400ee70 con 0x7f7c7409af30 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 5 monclient: authenticate success, global_id 4212 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient: _renew_subs 2024-09-16T07:18:53.014 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:53.015 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 1 -- 172.21.15.37:0/138285213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c740a2d20 con 0x7f7c7409af30 2024-09-16T07:18:53.015 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:53.015 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 1 -- 172.21.15.37:0/138285213 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400ca50 con 0x7f7c7409af30 2024-09-16T07:18:53.015 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:53.015 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:53.015 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c69ffb700 10 monclient: dump: 2024-09-16T07:18:53.015 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-09-16T07:18:53.015 INFO:teuthology.misc.health.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:53.015 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.015 INFO:teuthology.misc.health.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.016 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:53.016 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:53.016 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:53.016 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:53.016 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:53.016 INFO:teuthology.misc.health.smithi037.stderr: 2024-09-16T07:18:53.016 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient: _renew_subs 2024-09-16T07:18:53.016 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:53.016 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c7a09e700 1 -- 172.21.15.37:0/138285213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c741248c0 con 0x7f7c7409af30 2024-09-16T07:18:53.016 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.992+0000 7f7c6b7fe700 4 set_mon_vals no callback set 2024-09-16T07:18:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.996+0000 7f7c7a09e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.996+0000 7f7c7a09e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.996+0000 7f7c7a09e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.996+0000 7f7c7a09e700 1 -- 172.21.15.37:0/138285213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7409d720 con 0x7f7c7409af30 2024-09-16T07:18:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.996+0000 7f7c69ffb700 1 -- 172.21.15.37:0/138285213 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f7c6400cbb0 con 0x7f7c7409af30 2024-09-16T07:18:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.996+0000 7f7c69ffb700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7c58035600 0x7f7c58037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.996+0000 7f7c69ffb700 1 -- 172.21.15.37:0/138285213 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3993+0+0 (secure 0 0 0) 0x7f7c64047900 con 0x7f7c7409af30 2024-09-16T07:18:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.996+0000 7f7c7889b700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7c58035600 0x7f7c58037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.996+0000 7f7c7889b700 10 monclient: get_auth_request con 0x7f7c58035600 auth_method 0 2024-09-16T07:18:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:52.996+0000 7f7c7889b700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7c58035600 0x7f7c58037ac0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7c741221e0 tx=0x7f7c70007e00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.004+0000 7f7c69ffb700 1 -- 172.21.15.37:0/138285213 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7c6401bd80 con 0x7f7c7409af30 2024-09-16T07:18:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.004+0000 7f7c69ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.004+0000 7f7c69ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:53.182 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.180+0000 7f7c7a09e700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:53.182 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.180+0000 7f7c7a09e700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:53.182 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.180+0000 7f7c7a09e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:53.183 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.180+0000 7f7c7a09e700 1 -- 172.21.15.37:0/138285213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c740008d0 con 0x7f7c7409af30 2024-09-16T07:18:53.184 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c69ffb700 1 -- 172.21.15.37:0/138285213 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+10427 (secure 0 0 0) 0x7f7c64021020 con 0x7f7c7409af30 2024-09-16T07:18:53.184 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c69ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:53.184 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c69ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:53.184 INFO:teuthology.misc.health.smithi037.stdout: 2024-09-16T07:18:53.184 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":16,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","created":"2024-09-16T07:18:36.646941+0000","modified":"2024-09-16T07:18:49.404155+0000","last_up_change":"2024-09-16T07:18:46.372944+0000","last_in_change":"2024-09-16T07:18:38.297482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T07:18:46.507362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd764fc8-b5fb-40fe-a64f-ce7b8c1df525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6801","nonce":3137250735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6803","nonce":3137250735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6807","nonce":3137250735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6805","nonce":3137250735}]},"public_addr":"172.21.15.37:6801/3137250735","cluster_addr":"172.21.15.37:6803/3137250735","heartbeat_back_addr":"172.21.15.37:6807/3137250735","heartbeat_front_addr":"172.21.15.37:6805/3137250735","state":["exists","up"]},{"osd":1,"uuid":"52d577a5-08b7-452d-aa35-e1560215a09e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6809","nonce":810396637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6811","nonce":810396637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6815","nonce":810396637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6813","nonce":810396637}]},"public_addr":"172.21.15.37:6809/810396637","cluster_addr":"172.21.15.37:6811/810396637","heartbeat_back_addr":"172.21.15.37:6815/810396637","heartbeat_front_addr":"172.21.15.37:6813/810396637","state":["exists","up"]},{"osd":2,"uuid":"e3deb985-fb4e-42ee-8ed6-e6318cbca322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6817","nonce":265940208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6819","nonce":265940208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6823","nonce":265940208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6821","nonce":265940208}]},"public_addr":"172.21.15.37:6817/265940208","cluster_addr":"172.21.15.37:6819/265940208","heartbeat_back_addr":"172.21.15.37:6823/265940208","heartbeat_front_addr":"172.21.15.37:6821/265940208","state":["exists","up"]},{"osd":3,"uuid":"ab447540-8ed1-4a0c-8f6c-dfeea8af02ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6811","nonce":2588720019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6815","nonce":2588720019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6823","nonce":2588720019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6819","nonce":2588720019}]},"public_addr":"172.21.15.143:6811/2588720019","cluster_addr":"172.21.15.143:6815/2588720019","heartbeat_back_addr":"172.21.15.143:6823/2588720019","heartbeat_front_addr":"172.21.15.143:6819/2588720019","state":["exists","up"]},{"osd":4,"uuid":"be421fc0-1edf-4f20-b530-c9cf4eb20a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6810","nonce":3355421705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6814","nonce":3355421705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6822","nonce":3355421705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6818","nonce":3355421705}]},"public_addr":"172.21.15.143:6810/3355421705","cluster_addr":"172.21.15.143:6814/3355421705","heartbeat_back_addr":"172.21.15.143:6822/3355421705","heartbeat_front_addr":"172.21.15.143:6818/3355421705","state":["exists","up"]},{"osd":5,"uuid":"8c7725bd-5f56-4fa0-b455-6569e313f9fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6801","nonce":3921632375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6803","nonce":3921632375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6807","nonce":3921632375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6805","nonce":3921632375}]},"public_addr":"172.21.15.143:6801/3921632375","cluster_addr":"172.21.15.143:6803/3921632375","heartbeat_back_addr":"172.21.15.143:6807/3921632375","heartbeat_front_addr":"172.21.15.143:6805/3921632375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.492199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.565060+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.519201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.777218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.842229+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.307398+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:18:53.186 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 -- 172.21.15.37:0/138285213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7c58035600 msgr2=0x7f7c58037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7c58035600 0x7f7c58037ac0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7c741221e0 tx=0x7f7c70007e00 comp rx=0 tx=0).stop 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 10 monclient: shutdown 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 -- 172.21.15.37:0/138285213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409af30 msgr2=0x7f7c741ca550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409af30 0x7f7c741ca550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400e9d0 tx=0x7f7c6400ece0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 -- 172.21.15.37:0/138285213 shutdown_connections 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c7409fdb0 0x7f7c74120940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7c58035600 0x7f7c58037ac0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c7409f2c0 0x7f7c741caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 --2- 172.21.15.37:0/138285213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c7409af30 0x7f7c741ca550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 -- 172.21.15.37:0/138285213 >> 172.21.15.37:0/138285213 conn(0x7f7c740a4060 msgr2=0x7f7c741313c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:53.187 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 -- 172.21.15.37:0/138285213 shutdown_connections 2024-09-16T07:18:53.188 INFO:teuthology.misc.health.smithi037.stderr:2024-09-16T07:18:53.184+0000 7f7c7a09e700 1 -- 172.21.15.37:0/138285213 wait complete. 2024-09-16T07:18:53.197 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T07:18:53.197 INFO:tasks.ceph:Creating RBD pool 2024-09-16T07:18:53.197 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T07:18:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.308+0000 7faff9e5e700 10 monclient: get_monmap_and_config 2024-09-16T07:18:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.308+0000 7faff9e5e700 10 monclient: build_initial_monmap 2024-09-16T07:18:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.308+0000 7faff9e5e700 10 monclient: monmap: 2024-09-16T07:18:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:53.315332+0000 2024-09-16T07:18:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:53.315332+0000 2024-09-16T07:18:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:53.313 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 1 Processor -- start 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 1 -- start start 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient: init 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faff40a1090 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a0840 0x7faff413a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient(hunting): picked mon.noname-b con 0x7faff40a0840 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faff413ac90 0x7faff413d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient(hunting): picked mon.noname-c con 0x7faff413ac90 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff409fb30 con 0x7faff40a1090 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff409f850 con 0x7faff413ac90 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff40a4630 con 0x7faff40a0840 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient(hunting): authenticate will time out at 853.801884s 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a0840 0x7faff413a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a0840 0x7faff413a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39814/0 (socket says 172.21.15.37:39814) 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 1 -- 172.21.15.37:0/484804642 learned_addr learned my addr 172.21.15.37:0/484804642 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 10 monclient(hunting): get_auth_request con 0x7faff40a0840 auth_method 0 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff3fff700 1 --2- 172.21.15.37:0/484804642 >> v2:172.21.15.143:3300/0 conn(0x7faff413ac90 0x7faff413d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff3fff700 10 monclient(hunting): get_auth_request con 0x7faff413ac90 auth_method 0 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2024-09-16T07:18:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 1 -- 172.21.15.37:0/484804642 >> v2:172.21.15.143:3300/0 conn(0x7faff413ac90 msgr2=0x7faff413d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 1 --2- 172.21.15.37:0/484804642 >> v2:172.21.15.143:3300/0 conn(0x7faff413ac90 0x7faff413d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 1 -- 172.21.15.37:0/484804642 >> v1:172.21.15.143:6789/0 conn(0x7faff40a1090 legacy=0x7faff40a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 10 monclient: _finish_hunting 0 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 1 monclient: found mon.noname-b 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 1 -- 172.21.15.37:0/484804642 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff409f570 con 0x7faff40a0840 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7fafeaffd700 1 --2- 172.21.15.37:0/484804642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a0840 0x7faff413a750 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009910 tx=0x7fafdc009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10a8d7ab372d7a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 1 -- 172.21.15.37:0/484804642 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc011830 con 0x7faff40a0840 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 10 monclient: dump: 2024-09-16T07:18:53.333 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 10 monclient: _finish_auth 0 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 1 -- 172.21.15.37:0/484804642 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc011990 con 0x7faff40a0840 2024-09-16T07:18:53.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 1 -- 172.21.15.37:0/484804642 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc011d40 con 0x7faff40a0840 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 5 monclient: authenticate success, global_id 4214 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff9e5e700 4 set_mon_vals no callback set 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.312+0000 7faff2ffd700 10 monclient: dump: 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.335 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 10 monclient: shutdown 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 -- 172.21.15.37:0/484804642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a0840 msgr2=0x7faff413a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 --2- 172.21.15.37:0/484804642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a0840 0x7faff413a750 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009910 tx=0x7fafdc009c20 comp rx=0 tx=0).stop 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 -- 172.21.15.37:0/484804642 shutdown_connections 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 --2- 172.21.15.37:0/484804642 >> v2:172.21.15.143:3300/0 conn(0x7faff413ac90 0x7faff413d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 --2- 172.21.15.37:0/484804642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a0840 0x7faff413a750 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 -- 172.21.15.37:0/484804642 >> 172.21.15.37:0/484804642 conn(0x7faff40a8730 msgr2=0x7faff40a4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 -- 172.21.15.37:0/484804642 shutdown_connections 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 -- 172.21.15.37:0/484804642 wait complete. 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 10 monclient: build_initial_monmap 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 10 monclient: monmap: 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:53.337 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 Processor -- start 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 -- start start 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 10 monclient: init 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.316+0000 7faff9e5e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faff40a0840 0x7faff41ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient(hunting): picked mon.noname-a con 0x7faff40a0840 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a1090 0x7faff41398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient(hunting): picked mon.noname-c con 0x7faff40a1090 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff413ac90 0x7faff4139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient(hunting): picked mon.noname-b con 0x7faff413ac90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:53.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff41ca650 con 0x7faff40a0840 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff409dda0 con 0x7faff413ac90 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff4126270 con 0x7faff40a1090 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient(hunting): authenticate will time out at 853.808122s 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a1090 0x7faff41398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a1090 0x7faff41398c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39830/0 (socket says 172.21.15.37:39830) 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 1 -- 172.21.15.37:0/1926697741 learned_addr learned my addr 172.21.15.37:0/1926697741 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 10 monclient(hunting): get_auth_request con 0x7faff40a1090 auth_method 0 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:53.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff37fe700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faff40a0840 0x7faff41ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff37fe700 10 monclient(hunting): get_auth_request con 0x7faff40a0840 auth_method 0 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff3fff700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff413ac90 0x7faff4139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff3fff700 10 monclient(hunting): get_auth_request con 0x7faff413ac90 auth_method 0 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:53.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 1 -- 172.21.15.37:0/1926697741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff413ac90 msgr2=0x7faff4139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff413ac90 0x7faff4139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 1 -- 172.21.15.37:0/1926697741 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faff40a0840 msgr2=0x7faff41ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faff40a0840 0x7faff41ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:53.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 10 monclient: _finish_hunting 0 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 1 monclient: found mon.noname-c 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 1 -- 172.21.15.37:0/1926697741 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc009570 con 0x7faff40a1090 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7fafeaffd700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a1090 0x7faff41398c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faff40a4770 tx=0x7fafdc004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 1 -- 172.21.15.37:0/1926697741 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc004510 con 0x7faff40a1090 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 10 monclient: dump: 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.342 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 10 monclient: _finish_auth 0 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 1 -- 172.21.15.37:0/1926697741 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc0036c0 con 0x7faff40a1090 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 5 monclient: authenticate success, global_id 4217 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient: _renew_subs 2024-09-16T07:18:53.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 1 -- 172.21.15.37:0/1926697741 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff40a7530 con 0x7faff40a1090 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 1 -- 172.21.15.37:0/1926697741 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc003a70 con 0x7faff40a1090 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 10 monclient: dump: 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:53.344 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient: _renew_subs 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff27fc700 4 set_mon_vals no callback set 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 1 -- 172.21.15.37:0/1926697741 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff4125f90 con 0x7faff40a1090 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 1 -- 172.21.15.37:0/1926697741 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fafdc003bd0 con 0x7faff40a1090 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fafd4035600 0x7fafd4037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:53.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff9e5e700 1 -- 172.21.15.37:0/1926697741 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff41259d0 con 0x7faff40a1090 2024-09-16T07:18:53.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff0ff9700 1 -- 172.21.15.37:0/1926697741 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3993+0+0 (secure 0 0 0) 0x7fafdc021030 con 0x7faff40a1090 2024-09-16T07:18:53.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff37fe700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fafd4035600 0x7fafd4037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:53.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.320+0000 7faff37fe700 10 monclient: get_auth_request con 0x7fafd4035600 auth_method 0 2024-09-16T07:18:53.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.324+0000 7faff37fe700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fafd4035600 0x7fafd4037ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fafe4000f60 tx=0x7fafe4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:53.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.332+0000 7faff0ff9700 1 -- 172.21.15.37:0/1926697741 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fafdc015070 con 0x7faff40a1090 2024-09-16T07:18:53.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.332+0000 7faff0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:53.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.332+0000 7faff0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:53.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.496+0000 7faff9e5e700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T07:18:53.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.496+0000 7faff9e5e700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T07:18:53.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.496+0000 7faff9e5e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:53.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:53.496+0000 7faff9e5e700 1 -- 172.21.15.37:0/1926697741 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7faff40a1b50 con 0x7faff40a1090 2024-09-16T07:18:54.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.448+0000 7faff0ff9700 1 -- 172.21.15.37:0/1926697741 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7fafdc0340a0 con 0x7faff40a1090 2024-09-16T07:18:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.448+0000 7faff0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T07:18:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.448+0000 7faff0ff9700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T07:18:54.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 -- 172.21.15.37:0/1926697741 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fafd4035600 msgr2=0x7fafd4037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:54.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fafd4035600 0x7fafd4037ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fafe4000f60 tx=0x7fafe4009040 comp rx=0 tx=0).stop 2024-09-16T07:18:54.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 10 monclient: shutdown 2024-09-16T07:18:54.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 -- 172.21.15.37:0/1926697741 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a1090 msgr2=0x7faff41398c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a1090 0x7faff41398c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faff40a4770 tx=0x7fafdc004770 comp rx=0 tx=0).stop 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 -- 172.21.15.37:0/1926697741 shutdown_connections 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff413ac90 0x7faff4139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fafd4035600 0x7fafd4037ac0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fafe4000f60 tx=0x7fafe4009040 comp rx=0 tx=0).stop 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faff40a1090 0x7faff41398c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 --2- 172.21.15.37:0/1926697741 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faff40a0840 0x7faff41ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 -- 172.21.15.37:0/1926697741 >> 172.21.15.37:0/1926697741 conn(0x7faff40a8730 msgr2=0x7faff409c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 -- 172.21.15.37:0/1926697741 shutdown_connections 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.456+0000 7faff9e5e700 1 -- 172.21.15.37:0/1926697741 wait complete. 2024-09-16T07:18:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'rbd' created 2024-09-16T07:18:54.469 DEBUG:teuthology.orchestra.run.smithi037:> rbd --cluster ceph pool init rbd 2024-09-16T07:18:54.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.512+0000 7ff3da643780 10 monclient: get_monmap_and_config 2024-09-16T07:18:54.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.512+0000 7ff3da643780 10 monclient: build_initial_monmap 2024-09-16T07:18:54.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.512+0000 7ff3da643780 10 monclient: monmap: 2024-09-16T07:18:54.514 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:54.514 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:54.514 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:54.518474+0000 2024-09-16T07:18:54.515 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:54.518474+0000 2024-09-16T07:18:54.515 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:54.515 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:54.515 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:54.515 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:54.515 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:54.515 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:54.515 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:54.515 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:54.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.512+0000 7ff3da643780 1 Processor -- start 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 -- start start 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient: init 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient: _reopen_session rank -1 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 --2- >> v2:172.21.15.143:3300/0 conn(0x564956c3a4d0 0x564956c3a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient(hunting): picked mon.noname-c con 0x564956c3a4d0 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 --2- >> v2:172.21.15.37:3300/0 conn(0x564956ce8b30 0x564956ce8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient(hunting): picked mon.noname-a con 0x564956ce8b30 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient(hunting): picked mon.noname-a-legacy con 0x564956ce9480 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x564956ad5ca0 con 0x564956ce9480 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:54.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x564956b18540 con 0x564956ce8b30 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x564956b15b10 con 0x564956c3a4d0 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient(hunting): _renew_subs 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient(hunting): authenticate will time out at 855.004697s 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da3dc700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x564956ce9480 0x564956cf3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36572/0 (socket says 172.21.15.37:36572) 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x564956ce8b30 0x564956ce8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da3dc700 1 -- 172.21.15.37:0/4059062154 learned_addr learned my addr 172.21.15.37:0/4059062154 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d917a700 1 --2- 172.21.15.37:0/4059062154 >> v2:172.21.15.143:3300/0 conn(0x564956c3a4d0 0x564956c3a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 10 monclient(hunting): get_auth_request con 0x564956ce8b30 auth_method 0 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:54.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d917a700 10 monclient(hunting): get_auth_request con 0x564956c3a4d0 auth_method 0 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d917a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d917a700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d917a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d917a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d917a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d917a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 1 -- 172.21.15.37:0/4059062154 >> v2:172.21.15.143:3300/0 conn(0x564956c3a4d0 msgr2=0x564956c3a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:54.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 1 --2- 172.21.15.37:0/4059062154 >> v2:172.21.15.143:3300/0 conn(0x564956c3a4d0 0x564956c3a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 1 -- 172.21.15.37:0/4059062154 >> v1:172.21.15.37:6789/0 conn(0x564956ce9480 legacy=0x564956cf3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 1 -- 172.21.15.37:0/4059062154 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745577970 0 0) 0x564956ad5ca0 con 0x564956ce9480 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 10 monclient: _finish_hunting 0 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 1 monclient: found mon.noname-a 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 1 -- 172.21.15.37:0/4059062154 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564956aaebb0 con 0x564956ce8b30 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3d8979700 1 --2- 172.21.15.37:0/4059062154 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956ce8b30 0x564956ce8f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x564956c3b120 tx=0x7ff3cc00a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802f3c0ab9630990 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 1 -- 172.21.15.37:0/4059062154 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc00f790 con 0x564956ce8b30 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 10 monclient: dump: 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:54.527 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 10 monclient: _finish_auth 0 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 10 monclient: _check_auth_tickets 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 5 monclient: authenticate success, global_id 4218 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:54.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 1 -- 172.21.15.37:0/4059062154 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3cc00f8f0 con 0x564956ce8b30 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient: get_monmap_and_config success 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 4 set_mon_vals no callback set 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 1 -- 172.21.15.37:0/4059062154 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc00fc80 con 0x564956ce8b30 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3c7fff700 10 monclient: dump: 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:54.529 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient: shutdown 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 -- 172.21.15.37:0/4059062154 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956ce8b30 msgr2=0x564956ce8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 --2- 172.21.15.37:0/4059062154 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956ce8b30 0x564956ce8f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x564956c3b120 tx=0x7ff3cc00a780 comp rx=0 tx=0).stop 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 -- 172.21.15.37:0/4059062154 shutdown_connections 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 --2- 172.21.15.37:0/4059062154 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956ce8b30 0x564956ce8f40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 --2- 172.21.15.37:0/4059062154 >> v2:172.21.15.143:3300/0 conn(0x564956c3a4d0 0x564956c3a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:54.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 -- 172.21.15.37:0/4059062154 >> 172.21.15.37:0/4059062154 conn(0x564956c3b8b0 msgr2=0x564956c3bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 -- 172.21.15.37:0/4059062154 shutdown_connections 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 1 -- 172.21.15.37:0/4059062154 wait complete. 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.516+0000 7ff3da643780 10 monclient: build_initial_monmap 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient: monmap: 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:54.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 1 Processor -- start 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 1 -- start start 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient: init 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient: _reopen_session rank -1 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956c3a4d0 0x564956cfd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient(hunting): picked mon.noname-a con 0x564956c3a4d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564956ce8b30 0x564956cfdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient(hunting): picked mon.noname-c con 0x564956ce8b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564956ce9480 0x564956c54a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient(hunting): picked mon.noname-b con 0x564956ce9480 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x564956b11400 con 0x564956c3a4d0 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:54.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x564956acd9d0 con 0x564956ce9480 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x564956acd670 con 0x564956ce8b30 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient(hunting): _renew_subs 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient(hunting): authenticate will time out at 855.008994s 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956c3a4d0 0x564956cfd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956c3a4d0 0x564956cfd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36350/0 (socket says 172.21.15.37:36350) 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 1 -- 172.21.15.37:0/2944580263 learned_addr learned my addr 172.21.15.37:0/2944580263 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 10 monclient(hunting): get_auth_request con 0x564956c3a4d0 auth_method 0 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d8979700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564956ce8b30 0x564956cfdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d8979700 10 monclient(hunting): get_auth_request con 0x564956ce8b30 auth_method 0 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d8979700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:54.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da3dc700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564956ce9480 0x564956c54a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d8979700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d8979700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da3dc700 10 monclient(hunting): get_auth_request con 0x564956ce9480 auth_method 0 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da3dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da3dc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da3dc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d8979700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d8979700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d8979700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:54.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da3dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da3dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da3dc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 1 -- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564956ce8b30 msgr2=0x564956cfdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564956ce8b30 0x564956cfdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 1 -- 172.21.15.37:0/2944580263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564956ce9480 msgr2=0x564956c54a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564956ce9480 0x564956c54a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 10 monclient: _finish_hunting 0 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 1 monclient: found mon.noname-a 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc00a100 con 0x564956c3a4d0 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d8979700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564956ce8b30 0x564956cfdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:54.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da3dc700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564956ce9480 0x564956c54a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3d917a700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956c3a4d0 0x564956cfd750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80087c0 tx=0x7ff3c8008ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c800f040 con 0x564956c3a4d0 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 10 monclient: dump: 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:54.536 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 10 monclient: _finish_auth 0 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 10 monclient: _check_auth_tickets 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 5 monclient: authenticate success, global_id 4224 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c8013810 con 0x564956c3a4d0 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient: _renew_subs 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564956abbb00 con 0x564956c3a4d0 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8013ba0 con 0x564956c3a4d0 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:54.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c57fa700 10 monclient: dump: 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3c6ffd700 4 set_mon_vals no callback set 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient: _renew_subs 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:54.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.520+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564956aaef40 con 0x564956c3a4d0 2024-09-16T07:18:54.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.524+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7ff3c8005020 con 0x564956c3a4d0 2024-09-16T07:18:54.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.524+0000 7ff3c57fa700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff3b8033e40 0x7ff3b8036300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:54.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.524+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4335+0+0 (secure 0 0 0) 0x7ff3c80469a0 con 0x564956c3a4d0 2024-09-16T07:18:54.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.524+0000 7ff3da643780 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T07:18:54.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.524+0000 7ff3da643780 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T07:18:54.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.524+0000 7ff3da643780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:54.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.524+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x564956aae9a0 con 0x564956c3a4d0 2024-09-16T07:18:54.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.524+0000 7ff3d8979700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff3b8033e40 0x7ff3b8036300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:54.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.524+0000 7ff3d8979700 10 monclient: get_auth_request con 0x7ff3b8033e40 auth_method 0 2024-09-16T07:18:54.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:54.524+0000 7ff3d8979700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff3b8033e40 0x7ff3b8036300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc000f80 tx=0x7ff3cc007030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:55.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 141+0+0 (secure 0 0 0) 0x7ff3c80222a0 con 0x564956c3a4d0 2024-09-16T07:18:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3c57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T07:18:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3c57fa700 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T07:18:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3da643780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x564956ad4dd0 con 0x564956c3a4d0 2024-09-16T07:18:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3c80222a0 con 0x564956c3a4d0 2024-09-16T07:18:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3c57fa700 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T07:18:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3c6ffd700 10 monclient: _renew_subs 2024-09-16T07:18:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3c6ffd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3c6ffd700 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=18}) v3 -- 0x7ff3b0002ae0 con 0x564956c3a4d0 2024-09-16T07:18:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 8 ==== osd_map(18..18 src has 1..18) v4 ==== 674+0+0 (secure 0 0 0) 0x7ff3c8022d80 con 0x564956c3a4d0 2024-09-16T07:18:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3da643780 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x564956c55950 0x564956d06770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x564956d06cb0 con 0x564956c55950 2024-09-16T07:18:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3da3dc700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x564956c55950 0x564956d06770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.456+0000 7ff3da3dc700 10 monclient: get_auth_request con 0x564956c55950 auth_method 0 2024-09-16T07:18:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.460+0000 7ff3da3dc700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x564956c55950 0x564956d06770 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:55.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.464+0000 7ff3da3dc700 1 -- 172.21.15.37:0/2944580263 <== osd.1 v2:172.21.15.37:6808/810396637 1 ==== osd_op_reply(1 rbd_trash [create] v18'1 uv1 ondisk = 0) v8 ==== 153+0+0 (crc 0 0 0) 0x7ff3d000c8a0 con 0x564956c55950 2024-09-16T07:18:55.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.468+0000 7ff3da643780 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x564956d56930 0x564956d58dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:55.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.468+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x564956d59320 con 0x564956d56930 2024-09-16T07:18:55.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.468+0000 7ff3d917a700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x564956d56930 0x564956d58dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:55.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.468+0000 7ff3d917a700 10 monclient: get_auth_request con 0x564956d56930 auth_method 0 2024-09-16T07:18:55.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.468+0000 7ff3d917a700 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x564956d56930 0x564956d58dc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:55.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.468+0000 7ff3d917a700 1 -- 172.21.15.37:0/2944580263 <== osd.0 v2:172.21.15.37:6800/3137250735 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff3c8019070 con 0x564956d56930 2024-09-16T07:18:55.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.468+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) v8 -- 0x564956d55410 con 0x564956d56930 2024-09-16T07:18:55.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.472+0000 7ff3d917a700 1 -- 172.21.15.37:0/2944580263 <== osd.0 v2:172.21.15.37:6800/3137250735 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff3c8019070 con 0x564956d56930 2024-09-16T07:18:55.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.472+0000 7ff3c77fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:55.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:55.472+0000 7ff3c77fe700 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) v4 -- 0x7ff3b4001f20 con 0x564956c3a4d0 2024-09-16T07:18:56.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:56.460+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 9 ==== pool_op_reply(tid 4 (0) Success v19) v1 ==== 55+0+0 (secure 0 0 0) 0x7ff3c803e050 con 0x564956c3a4d0 2024-09-16T07:18:56.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:56.460+0000 7ff3c57fa700 10 monclient: _renew_subs 2024-09-16T07:18:56.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:56.460+0000 7ff3c57fa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:56.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:56.460+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=19}) v3 -- 0x7ff3b8039f90 con 0x564956c3a4d0 2024-09-16T07:18:56.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:56.460+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 10 ==== osd_map(19..19 src has 1..19) v4 ==== 626+0+0 (secure 0 0 0) 0x7ff3c802d020 con 0x564956c3a4d0 2024-09-16T07:18:56.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:56.464+0000 7ff3c6ffd700 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x7ff3b0005620 con 0x564956d56930 2024-09-16T07:18:56.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:56.464+0000 7ff3d917a700 1 -- 172.21.15.37:0/2944580263 <== osd.0 v2:172.21.15.37:6800/3137250735 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v19'1 uv1 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7ff3c8019070 con 0x564956d56930 2024-09-16T07:18:56.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:56.464+0000 7ff3c77fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:56.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:56.464+0000 7ff3c77fe700 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v19) v4 -- 0x7ff3b40033a0 con 0x564956c3a4d0 2024-09-16T07:18:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.468+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 11 ==== pool_op_reply(tid 6 (0) Success v20) v1 ==== 43+0+0 (secure 0 0 0) 0x7ff3c8011ab0 con 0x564956c3a4d0 2024-09-16T07:18:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.468+0000 7ff3c57fa700 10 monclient: _renew_subs 2024-09-16T07:18:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.468+0000 7ff3c57fa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.468+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=20}) v3 -- 0x7ff3b803a120 con 0x564956c3a4d0 2024-09-16T07:18:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.468+0000 7ff3c57fa700 1 -- 172.21.15.37:0/2944580263 <== mon.0 v2:172.21.15.37:3300/0 12 ==== osd_map(20..20 src has 1..20) v4 ==== 654+0+0 (secure 0 0 0) 0x7ff3c8046be0 con 0x564956c3a4d0 2024-09-16T07:18:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.468+0000 7ff3c6ffd700 1 -- 172.21.15.37:0/2944580263 --> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) v8 -- 0x7ff3b0005a10 con 0x564956d56930 2024-09-16T07:18:57.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3d917a700 1 -- 172.21.15.37:0/2944580263 <== osd.0 v2:172.21.15.37:6800/3137250735 4 ==== osd_op_reply(7 rbd_info [write 0~19] v20'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff3c8019070 con 0x564956d56930 2024-09-16T07:18:57.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x564956d56930 msgr2=0x564956d58dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x564956d56930 0x564956d58dc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x564956c55950 msgr2=0x564956d06770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x564956c55950 0x564956d06770 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff3b8033e40 msgr2=0x7ff3b8036300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff3b8033e40 0x7ff3b8036300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc000f80 tx=0x7ff3cc007030 comp rx=0 tx=0).stop 2024-09-16T07:18:57.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 10 monclient: shutdown 2024-09-16T07:18:57.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:57.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da3dc700 1 -- 172.21.15.37:0/2944580263 reap_dead start 2024-09-16T07:18:57.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956c3a4d0 msgr2=0x564956cfd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956c3a4d0 0x564956cfd750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80087c0 tx=0x7ff3c8008ad0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 shutdown_connections 2024-09-16T07:18:57.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 --2- 172.21.15.37:0/2944580263 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564956c3a4d0 0x564956cfd750 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 >> 172.21.15.37:0/2944580263 conn(0x564956c3b8b0 msgr2=0x564956cee100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:57.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 shutdown_connections 2024-09-16T07:18:57.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.472+0000 7ff3da643780 1 -- 172.21.15.37:0/2944580263 wait complete. 2024-09-16T07:18:57.478 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T07:18:57.479 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T07:18:57.479 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T07:18:57.479 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T07:18:57.480 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T07:18:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.592+0000 7f9e84953700 10 monclient: get_monmap_and_config 2024-09-16T07:18:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.592+0000 7f9e84953700 10 monclient: build_initial_monmap 2024-09-16T07:18:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.592+0000 7f9e84953700 10 monclient: monmap: 2024-09-16T07:18:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:57.598713+0000 2024-09-16T07:18:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:57.598713+0000 2024-09-16T07:18:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:57.595 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 1 Processor -- start 2024-09-16T07:18:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 1 -- start start 2024-09-16T07:18:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient: init 2024-09-16T07:18:57.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9e80131a70 0x7f9e8009c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient(hunting): picked mon.noname-c con 0x7f9e80131a70 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e80131e50 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8009a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient(hunting): picked mon.noname-b con 0x7f9e80132230 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8009b590 con 0x7f9e80131e50 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8009b870 con 0x7f9e80131a70 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8009bb50 con 0x7f9e80132230 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient(hunting): authenticate will time out at 858.086199s 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e77fff700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9e80131e50 0x7f9e800999e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36580/0 (socket says 172.21.15.37:36580) 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8009a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e77fff700 1 -- 172.21.15.37:0/1408516845 learned_addr learned my addr 172.21.15.37:0/1408516845 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 10 monclient(hunting): get_auth_request con 0x7f9e80132230 auth_method 0 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7f7fe700 1 --2- 172.21.15.37:0/1408516845 >> v2:172.21.15.143:3300/0 conn(0x7f9e80131a70 0x7f9e8009c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7f7fe700 10 monclient(hunting): get_auth_request con 0x7f9e80131a70 auth_method 0 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 1 -- 172.21.15.37:0/1408516845 >> v2:172.21.15.143:3300/0 conn(0x7f9e80131a70 msgr2=0x7f9e8009c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 1 --2- 172.21.15.37:0/1408516845 >> v2:172.21.15.143:3300/0 conn(0x7f9e80131a70 0x7f9e8009c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 1 -- 172.21.15.37:0/1408516845 >> v1:172.21.15.37:6789/0 conn(0x7f9e80131e50 legacy=0x7f9e800999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 10 monclient: _finish_hunting 0 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 1 monclient: found mon.noname-b 2024-09-16T07:18:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 1 -- 172.21.15.37:0/1408516845 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416529528 0 0) 0x7f9e8009b590 con 0x7f9e80131e50 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 1 -- 172.21.15.37:0/1408516845 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8009e350 con 0x7f9e80132230 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7ffff700 1 --2- 172.21.15.37:0/1408516845 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8009a0f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009d90 tx=0x7f9e7800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=858babc39347058a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 1 -- 172.21.15.37:0/1408516845 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800e070 con 0x7f9e80132230 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 10 monclient: dump: 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 10 monclient: _finish_auth 0 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 1 -- 172.21.15.37:0/1408516845 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7800bbd0 con 0x7f9e80132230 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 1 -- 172.21.15.37:0/1408516845 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78012430 con 0x7f9e80132230 2024-09-16T07:18:57.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 5 monclient: authenticate success, global_id 4223 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e84953700 4 set_mon_vals no callback set 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.596+0000 7f9e7effd700 10 monclient: dump: 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:57.604 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:57.605 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:57.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 10 monclient: shutdown 2024-09-16T07:18:57.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:57.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 1 -- 172.21.15.37:0/1408516845 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 msgr2=0x7f9e8009a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 1 --2- 172.21.15.37:0/1408516845 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8009a0f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009d90 tx=0x7f9e7800b330 comp rx=0 tx=0).stop 2024-09-16T07:18:57.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 1 -- 172.21.15.37:0/1408516845 shutdown_connections 2024-09-16T07:18:57.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 1 --2- 172.21.15.37:0/1408516845 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8009a0f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 1 --2- 172.21.15.37:0/1408516845 >> v2:172.21.15.143:3300/0 conn(0x7f9e80131a70 0x7f9e8009c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 1 -- 172.21.15.37:0/1408516845 >> 172.21.15.37:0/1408516845 conn(0x7f9e800a3f60 msgr2=0x7f9e800a4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 1 -- 172.21.15.37:0/1408516845 shutdown_connections 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 1 -- 172.21.15.37:0/1408516845 wait complete. 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 10 monclient: build_initial_monmap 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 10 monclient: monmap: 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:57.612 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:57.612 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:57.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 1 Processor -- start 2024-09-16T07:18:57.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 1 -- start start 2024-09-16T07:18:57.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.600+0000 7f9e84953700 10 monclient: init 2024-09-16T07:18:57.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:57.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:18:57.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80131a70 0x7f9e801311c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:57.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient(hunting): picked mon.noname-b con 0x7f9e80131a70 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:57.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e80131e50 0x7f9e8011b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient(hunting): picked mon.noname-a con 0x7f9e80131e50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8011beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient(hunting): picked mon.noname-c con 0x7f9e80132230 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e800a0420 con 0x7f9e80131e50 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8009b150 con 0x7f9e80131a70 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e80099f80 con 0x7f9e80132230 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient(hunting): authenticate will time out at 858.092423s 2024-09-16T07:18:57.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8011beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e77fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e80131e50 0x7f9e8011b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80131a70 0x7f9e801311c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8011beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39854/0 (socket says 172.21.15.37:39854) 2024-09-16T07:18:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 1 -- 172.21.15.37:0/2087667716 learned_addr learned my addr 172.21.15.37:0/2087667716 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e77fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e80131e50 0x7f9e8011b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36360/0 (socket says 172.21.15.37:36360) 2024-09-16T07:18:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 10 monclient(hunting): get_auth_request con 0x7f9e80132230 auth_method 0 2024-09-16T07:18:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:57.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:57.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:57.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7f7fe700 10 monclient(hunting): get_auth_request con 0x7f9e80131a70 auth_method 0 2024-09-16T07:18:57.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:57.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:57.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:57.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e77fff700 10 monclient(hunting): get_auth_request con 0x7f9e80131e50 auth_method 0 2024-09-16T07:18:57.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:57.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e77fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:57.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-09-16T07:18:57.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 1 -- 172.21.15.37:0/2087667716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80131a70 msgr2=0x7f9e801311c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80131a70 0x7f9e801311c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 1 -- 172.21.15.37:0/2087667716 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e80131e50 msgr2=0x7f9e8011b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e80131e50 0x7f9e8011b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 10 monclient: _finish_hunting 0 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 1 monclient: found mon.noname-c 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 1 -- 172.21.15.37:0/2087667716 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e78009a70 con 0x7f9e80132230 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7ffff700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8011beb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e8006be10 tx=0x7f9e78004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 1 -- 172.21.15.37:0/2087667716 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78012430 con 0x7f9e80132230 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:57.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 10 monclient: dump: 2024-09-16T07:18:57.618 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:57.618 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:57.618 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.618 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.618 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:57.618 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:57.618 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:57.618 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:57.618 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:57.618 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:57.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 10 monclient: _finish_auth 0 2024-09-16T07:18:57.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 10 monclient: _check_auth_tickets 2024-09-16T07:18:57.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:57.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 1 -- 172.21.15.37:0/2087667716 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e78020070 con 0x7f9e80132230 2024-09-16T07:18:57.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:57.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 1 -- 172.21.15.37:0/2087667716 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78004040 con 0x7f9e80132230 2024-09-16T07:18:57.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 5 monclient: authenticate success, global_id 4226 2024-09-16T07:18:57.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:57.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:57.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7e7fc700 4 set_mon_vals no callback set 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 10 monclient: dump: 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:57.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient: _renew_subs 2024-09-16T07:18:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 1 -- 172.21.15.37:0/2087667716 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8011f8d0 con 0x7f9e80132230 2024-09-16T07:18:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient: _renew_subs 2024-09-16T07:18:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e84953700 1 -- 172.21.15.37:0/2087667716 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8011f5f0 con 0x7f9e80132230 2024-09-16T07:18:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.604+0000 7f9e7cff9700 1 -- 172.21.15.37:0/2087667716 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f9e780041a0 con 0x7f9e80132230 2024-09-16T07:18:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.608+0000 7f9e7cff9700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9e60035600 0x7f9e60037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.608+0000 7f9e7cff9700 1 -- 172.21.15.37:0/2087667716 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7f9e78022070 con 0x7f9e80132230 2024-09-16T07:18:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.608+0000 7f9e84953700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.608+0000 7f9e84953700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.608+0000 7f9e84953700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.608+0000 7f9e84953700 1 -- 172.21.15.37:0/2087667716 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8011f030 con 0x7f9e80132230 2024-09-16T07:18:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.608+0000 7f9e7f7fe700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9e60035600 0x7f9e60037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:57.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.608+0000 7f9e7f7fe700 10 monclient: get_auth_request con 0x7f9e60035600 auth_method 0 2024-09-16T07:18:57.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.608+0000 7f9e7f7fe700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9e60035600 0x7f9e60037ac0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009a00 tx=0x7f9e70008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:57.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.616+0000 7f9e7cff9700 1 -- 172.21.15.37:0/2087667716 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9e7802c020 con 0x7f9e80132230 2024-09-16T07:18:57.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.616+0000 7f9e7cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:57.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.616+0000 7f9e7cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:57.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.832+0000 7f9e84953700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T07:18:57.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.832+0000 7f9e84953700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T07:18:57.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.832+0000 7f9e84953700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:57.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.832+0000 7f9e84953700 1 -- 172.21.15.37:0/2087667716 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f9e8009b590 con 0x7f9e80132230 2024-09-16T07:18:57.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.836+0000 7f9e7cff9700 1 -- 172.21.15.37:0/2087667716 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+83121 (secure 0 0 0) 0x7f9e78011570 con 0x7f9e80132230 2024-09-16T07:18:57.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.836+0000 7f9e7cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T07:18:57.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.836+0000 7f9e7cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:57.840 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:57.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 -- 172.21.15.37:0/2087667716 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9e60035600 msgr2=0x7f9e60037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9e60035600 0x7f9e60037ac0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009a00 tx=0x7f9e70008040 comp rx=0 tx=0).stop 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 10 monclient: shutdown 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 -- 172.21.15.37:0/2087667716 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 msgr2=0x7f9e8011beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8011beb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e8006be10 tx=0x7f9e78004af0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 -- 172.21.15.37:0/2087667716 shutdown_connections 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9e60035600 0x7f9e60037ac0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e80132230 0x7f9e8011beb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e80131e50 0x7f9e8011b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 --2- 172.21.15.37:0/2087667716 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80131a70 0x7f9e801311c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 -- 172.21.15.37:0/2087667716 >> 172.21.15.37:0/2087667716 conn(0x7f9e800a3f60 msgr2=0x7f9e8009ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:57.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 -- 172.21.15.37:0/2087667716 shutdown_connections 2024-09-16T07:18:57.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.844+0000 7f9e84953700 1 -- 172.21.15.37:0/2087667716 wait complete. 2024-09-16T07:18:57.860 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":3705696220},{"type":"v1","addr":"172.21.15.143:6825","nonce":3705696220}]},"active_addr":"172.21.15.143:6825/3705696220","active_change":"2024-09-16T07:18:42.430524+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":580768763}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":3621073495}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":1750256181}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":4237163555}]}]} 2024-09-16T07:18:57.863 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T07:18:57.863 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T07:18:57.863 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T07:18:57.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.972+0000 7fed3359e700 10 monclient: get_monmap_and_config 2024-09-16T07:18:57.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.972+0000 7fed3359e700 10 monclient: build_initial_monmap 2024-09-16T07:18:57.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.972+0000 7fed3359e700 10 monclient: monmap: 2024-09-16T07:18:57.974 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:57.974 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:57.974 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:57.978296+0000 2024-09-16T07:18:57.975 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:57.978296+0000 2024-09-16T07:18:57.975 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:57.975 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:57.975 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:57.975 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:57.975 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:57.975 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:57.975 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:57.975 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:57.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.972+0000 7fed3359e700 1 Processor -- start 2024-09-16T07:18:57.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.972+0000 7fed3359e700 1 -- start start 2024-09-16T07:18:57.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.972+0000 7fed3359e700 10 monclient: init 2024-09-16T07:18:57.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:57.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-16T07:18:57.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fed340a2750 0x7fed340999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:57.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient(hunting): picked mon.noname-a con 0x7fed340a2750 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:57.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed34071880 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed340a31e0 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed340a2b30 con 0x7fed34071880 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3409c3e0 con 0x7fed340a31e0 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed3409d0e0 con 0x7fed340a2750 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient(hunting): authenticate will time out at 858.464172s 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fed340a2750 0x7fed340999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed29d9b700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fed34071880 0x7fed34099f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36586/0 (socket says 172.21.15.37:36586) 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed29d9b700 1 -- 172.21.15.37:0/2389840821 learned_addr learned my addr 172.21.15.37:0/2389840821 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 10 monclient(hunting): get_auth_request con 0x7fed340a2750 auth_method 0 2024-09-16T07:18:57.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:57.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:57.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:57.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:57.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:57.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:57.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-09-16T07:18:57.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 1 -- 172.21.15.37:0/2389840821 >> v1:172.21.15.143:6789/0 conn(0x7fed340a31e0 legacy=0x7fed34139050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 1 -- 172.21.15.37:0/2389840821 >> v1:172.21.15.37:6789/0 conn(0x7fed34071880 legacy=0x7fed34099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 10 monclient: _finish_hunting 0 2024-09-16T07:18:57.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 1 monclient: found mon.noname-a 2024-09-16T07:18:57.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:57.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:57.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 1 -- 172.21.15.37:0/2389840821 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3409ab10 con 0x7fed340a2750 2024-09-16T07:18:57.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3259c700 1 --2- 172.21.15.37:0/2389840821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a2750 0x7fed340999e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fed3409d220 tx=0x7fed24009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5fa81a72ac4a6b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:57.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 1 -- 172.21.15.37:0/2389840821 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed24004230 con 0x7fed340a2750 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 10 monclient: dump: 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:57.991 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 10 monclient: _finish_auth 0 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 10 monclient: _check_auth_tickets 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 5 monclient: authenticate success, global_id 4233 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 1 -- 172.21.15.37:0/2389840821 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed24004390 con 0x7fed340a2750 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 4 set_mon_vals no callback set 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 1 -- 172.21.15.37:0/2389840821 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed240036c0 con 0x7fed340a2750 2024-09-16T07:18:57.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed31d9b700 10 monclient: dump: 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:57.993 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient: shutdown 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 -- 172.21.15.37:0/2389840821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a2750 msgr2=0x7fed340999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 --2- 172.21.15.37:0/2389840821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a2750 0x7fed340999e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fed3409d220 tx=0x7fed24009b30 comp rx=0 tx=0).stop 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 -- 172.21.15.37:0/2389840821 shutdown_connections 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 --2- 172.21.15.37:0/2389840821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a2750 0x7fed340999e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 -- 172.21.15.37:0/2389840821 >> 172.21.15.37:0/2389840821 conn(0x7fed340a3ea0 msgr2=0x7fed340a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 -- 172.21.15.37:0/2389840821 shutdown_connections 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 -- 172.21.15.37:0/2389840821 wait complete. 2024-09-16T07:18:57.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient: build_initial_monmap 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient: monmap: 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:57.995 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 Processor -- start 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 1 -- start start 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.976+0000 7fed3359e700 10 monclient: init 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34071880 0x7fed341ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient(hunting): picked mon.noname-b con 0x7fed34071880 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a2750 0x7fed341c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient(hunting): picked mon.noname-c con 0x7fed340a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a31e0 0x7fed341c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:57.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient(hunting): picked mon.noname-a con 0x7fed340a31e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed341a92b0 con 0x7fed340a31e0 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed341a98d0 con 0x7fed34071880 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed340724c0 con 0x7fed340a2750 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient(hunting): authenticate will time out at 858.468311s 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed29d9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a2750 0x7fed341c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a31e0 0x7fed341c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34071880 0x7fed341ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:57.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34071880 0x7fed341ca260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.37:51296/0 (socket says 172.21.15.37:51296) 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 1 -- 172.21.15.37:0/2748203037 learned_addr learned my addr 172.21.15.37:0/2748203037 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 10 monclient(hunting): get_auth_request con 0x7fed34071880 auth_method 0 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a31e0 0x7fed341c6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36368/0 (socket says 172.21.15.37:36368) 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 10 monclient(hunting): get_auth_request con 0x7fed340a31e0 auth_method 0 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed29d9b700 10 monclient(hunting): get_auth_request con 0x7fed340a2750 auth_method 0 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed29d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed29d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:57.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed29d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed29d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed29d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed29d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 1 -- 172.21.15.37:0/2748203037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a2750 msgr2=0x7fed341c6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a2750 0x7fed341c6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:57.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 1 -- 172.21.15.37:0/2748203037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34071880 msgr2=0x7fed341ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34071880 0x7fed341ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 1 -- 172.21.15.37:0/2748203037 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed240094b0 con 0x7fed340a31e0 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed29d9b700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a2750 0x7fed341c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3259c700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34071880 0x7fed341ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed32d9d700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a31e0 0x7fed341c6ce0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fed2c00cb90 tx=0x7fed2c00cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 1 -- 172.21.15.37:0/2748203037 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c003c20 con 0x7fed340a31e0 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:58.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 10 monclient: dump: 2024-09-16T07:18:58.001 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.001 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.001 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.001 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.001 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.001 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.001 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.001 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.001 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 1 -- 172.21.15.37:0/2748203037 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c003d80 con 0x7fed340a31e0 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 5 monclient: authenticate success, global_id 4239 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient: _renew_subs 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 1 -- 172.21.15.37:0/2748203037 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed34070f30 con 0x7fed340a31e0 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 1 -- 172.21.15.37:0/2748203037 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c00aa80 con 0x7fed340a31e0 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 10 monclient: dump: 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3159a700 4 set_mon_vals no callback set 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient: _renew_subs 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed3359e700 1 -- 172.21.15.37:0/2748203037 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed341cacc0 con 0x7fed340a31e0 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.980+0000 7fed2b7fe700 1 -- 172.21.15.37:0/2748203037 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fed2c00abe0 con 0x7fed340a31e0 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.984+0000 7fed3359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.984+0000 7fed3359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.984+0000 7fed3359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.984+0000 7fed3359e700 1 -- 172.21.15.37:0/2748203037 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3409f1e0 con 0x7fed340a31e0 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.984+0000 7fed2b7fe700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fed14035600 0x7fed14037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.984+0000 7fed2b7fe700 1 -- 172.21.15.37:0/2748203037 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7fed2c047a20 con 0x7fed340a31e0 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.984+0000 7fed3259c700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fed14035600 0x7fed14037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.984+0000 7fed3259c700 10 monclient: get_auth_request con 0x7fed14035600 auth_method 0 2024-09-16T07:18:58.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.984+0000 7fed3259c700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fed14035600 0x7fed14037ac0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fed24000c00 tx=0x7fed24008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.992+0000 7fed2b7fe700 1 -- 172.21.15.37:0/2748203037 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fed2c0198f0 con 0x7fed340a31e0 2024-09-16T07:18:58.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.992+0000 7fed2b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:58.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:57.992+0000 7fed2b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:58.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.164+0000 7fed3359e700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:58.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.164+0000 7fed3359e700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:58.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.164+0000 7fed3359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.164+0000 7fed3359e700 1 -- 172.21.15.37:0/2748203037 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed340a0a00 con 0x7fed340a31e0 2024-09-16T07:18:58.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed2b7fe700 1 -- 172.21.15.37:0/2748203037 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+12276 (secure 0 0 0) 0x7fed2c019300 con 0x7fed340a31e0 2024-09-16T07:18:58.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed2b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:58.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed2b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:58.168 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:58.168 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":20,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","created":"2024-09-16T07:18:36.646941+0000","modified":"2024-09-16T07:18:57.461117+0000","last_up_change":"2024-09-16T07:18:46.372944+0000","last_in_change":"2024-09-16T07:18:38.297482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T07:18:46.507362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T07:18:53.543683+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd764fc8-b5fb-40fe-a64f-ce7b8c1df525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6801","nonce":3137250735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6803","nonce":3137250735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6807","nonce":3137250735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6805","nonce":3137250735}]},"public_addr":"172.21.15.37:6801/3137250735","cluster_addr":"172.21.15.37:6803/3137250735","heartbeat_back_addr":"172.21.15.37:6807/3137250735","heartbeat_front_addr":"172.21.15.37:6805/3137250735","state":["exists","up"]},{"osd":1,"uuid":"52d577a5-08b7-452d-aa35-e1560215a09e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6809","nonce":810396637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6811","nonce":810396637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6815","nonce":810396637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6813","nonce":810396637}]},"public_addr":"172.21.15.37:6809/810396637","cluster_addr":"172.21.15.37:6811/810396637","heartbeat_back_addr":"172.21.15.37:6815/810396637","heartbeat_front_addr":"172.21.15.37:6813/810396637","state":["exists","up"]},{"osd":2,"uuid":"e3deb985-fb4e-42ee-8ed6-e6318cbca322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6817","nonce":265940208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6819","nonce":265940208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6823","nonce":265940208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6821","nonce":265940208}]},"public_addr":"172.21.15.37:6817/265940208","cluster_addr":"172.21.15.37:6819/265940208","heartbeat_back_addr":"172.21.15.37:6823/265940208","heartbeat_front_addr":"172.21.15.37:6821/265940208","state":["exists","up"]},{"osd":3,"uuid":"ab447540-8ed1-4a0c-8f6c-dfeea8af02ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6811","nonce":2588720019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6815","nonce":2588720019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6823","nonce":2588720019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6819","nonce":2588720019}]},"public_addr":"172.21.15.143:6811/2588720019","cluster_addr":"172.21.15.143:6815/2588720019","heartbeat_back_addr":"172.21.15.143:6823/2588720019","heartbeat_front_addr":"172.21.15.143:6819/2588720019","state":["exists","up"]},{"osd":4,"uuid":"be421fc0-1edf-4f20-b530-c9cf4eb20a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6810","nonce":3355421705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6814","nonce":3355421705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6822","nonce":3355421705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6818","nonce":3355421705}]},"public_addr":"172.21.15.143:6810/3355421705","cluster_addr":"172.21.15.143:6814/3355421705","heartbeat_back_addr":"172.21.15.143:6822/3355421705","heartbeat_front_addr":"172.21.15.143:6818/3355421705","state":["exists","up"]},{"osd":5,"uuid":"8c7725bd-5f56-4fa0-b455-6569e313f9fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6801","nonce":3921632375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6803","nonce":3921632375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6807","nonce":3921632375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6805","nonce":3921632375}]},"public_addr":"172.21.15.143:6801/3921632375","cluster_addr":"172.21.15.143:6803/3921632375","heartbeat_back_addr":"172.21.15.143:6807/3921632375","heartbeat_front_addr":"172.21.15.143:6805/3921632375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.492199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.565060+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.519201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.777218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.842229+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.307398+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:18:58.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 -- 172.21.15.37:0/2748203037 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fed14035600 msgr2=0x7fed14037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fed14035600 0x7fed14037ac0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fed24000c00 tx=0x7fed24008030 comp rx=0 tx=0).stop 2024-09-16T07:18:58.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 10 monclient: shutdown 2024-09-16T07:18:58.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 -- 172.21.15.37:0/2748203037 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a31e0 msgr2=0x7fed341c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a31e0 0x7fed341c6ce0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fed2c00cb90 tx=0x7fed2c00cea0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 -- 172.21.15.37:0/2748203037 shutdown_connections 2024-09-16T07:18:58.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed340a31e0 0x7fed341c6ce0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fed14035600 0x7fed14037ac0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a2750 0x7fed341c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 --2- 172.21.15.37:0/2748203037 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed34071880 0x7fed341ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 -- 172.21.15.37:0/2748203037 >> 172.21.15.37:0/2748203037 conn(0x7fed340a3ea0 msgr2=0x7fed3409ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 -- 172.21.15.37:0/2748203037 shutdown_connections 2024-09-16T07:18:58.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.168+0000 7fed3359e700 1 -- 172.21.15.37:0/2748203037 wait complete. 2024-09-16T07:18:58.183 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T07:18:58.183 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T07:18:58.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.292+0000 7fa2bb59e700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.292+0000 7fa2bb59e700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.292+0000 7fa2bb59e700 10 monclient: monmap: 2024-09-16T07:18:58.292 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.292 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.293 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.296768+0000 2024-09-16T07:18:58.293 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.296768+0000 2024-09-16T07:18:58.293 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.293 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.293 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.293 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.293 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.293 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.293 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.293 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.292+0000 7fa2bb59e700 1 Processor -- start 2024-09-16T07:18:58.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.292+0000 7fa2bb59e700 1 -- start start 2024-09-16T07:18:58.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.292+0000 7fa2bb59e700 10 monclient: init 2024-09-16T07:18:58.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T07:18:58.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc06d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa2bc06d430 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2bc0a1050 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:58.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa2bc06dd50 0x7fa2bc132890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa2bc06dd50 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:58.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc09ee80 con 0x7fa2bc0a1050 2024-09-16T07:18:58.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2bc06f980 con 0x7fa2bc06dd50 2024-09-16T07:18:58.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0a0dd0 con 0x7fa2bc06d430 2024-09-16T07:18:58.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient(hunting): authenticate will time out at 858.784412s 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc06d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bad9d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa2bc06dd50 0x7fa2bc132890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc06d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39880/0 (socket says 172.21.15.37:39880) 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 1 -- 172.21.15.37:0/21949405 learned_addr learned my addr 172.21.15.37:0/21949405 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 10 monclient(hunting): get_auth_request con 0x7fa2bc06d430 auth_method 0 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bad9d700 10 monclient(hunting): get_auth_request con 0x7fa2bc06dd50 auth_method 0 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 1 -- 172.21.15.37:0/21949405 >> v2:172.21.15.37:3300/0 conn(0x7fa2bc06dd50 msgr2=0x7fa2bc132890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 1 --2- 172.21.15.37:0/21949405 >> v2:172.21.15.37:3300/0 conn(0x7fa2bc06dd50 0x7fa2bc132890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 1 -- 172.21.15.37:0/21949405 >> v1:172.21.15.143:6789/0 conn(0x7fa2bc0a1050 legacy=0x7fa2bc132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 1 monclient: found mon.noname-b 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 1 -- 172.21.15.37:0/21949405 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc06f0e0 con 0x7fa2bc06d430 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2ba59c700 1 --2- 172.21.15.37:0/21949405 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc06d810 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009910 tx=0x7fa2ac009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30c3ac9139deca3b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 1 -- 172.21.15.37:0/21949405 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac011830 con 0x7fa2bc06d430 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 10 monclient: dump: 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.303 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 5 monclient: authenticate success, global_id 4232 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 1 -- 172.21.15.37:0/21949405 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac011990 con 0x7fa2bc06d430 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 4 set_mon_vals no callback set 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 1 -- 172.21.15.37:0/21949405 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac011d40 con 0x7fa2bc06d430 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2b959a700 10 monclient: dump: 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.305 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient: shutdown 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 -- 172.21.15.37:0/21949405 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 msgr2=0x7fa2bc06d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 --2- 172.21.15.37:0/21949405 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc06d810 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009910 tx=0x7fa2ac009c20 comp rx=0 tx=0).stop 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 -- 172.21.15.37:0/21949405 shutdown_connections 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 --2- 172.21.15.37:0/21949405 >> v2:172.21.15.37:3300/0 conn(0x7fa2bc06dd50 0x7fa2bc132890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 --2- 172.21.15.37:0/21949405 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc06d810 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 -- 172.21.15.37:0/21949405 >> 172.21.15.37:0/21949405 conn(0x7fa2bc0a14c0 msgr2=0x7fa2bc0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 -- 172.21.15.37:0/21949405 shutdown_connections 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 1 -- 172.21.15.37:0/21949405 wait complete. 2024-09-16T07:18:58.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.296+0000 7fa2bb59e700 10 monclient: monmap: 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:58.307 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 1 Processor -- start 2024-09-16T07:18:58.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 1 -- start start 2024-09-16T07:18:58.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient: init 2024-09-16T07:18:58.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:18:58.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc1ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa2bc06d430 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2bc06dd50 0x7fa2bc1cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa2bc06dd50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2bc121c20 0x7fa2bc1220a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa2bc121c20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc072660 con 0x7fa2bc121c20 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1244c0 con 0x7fa2bc06dd50 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc0a3030 con 0x7fa2bc06d430 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient(hunting): authenticate will time out at 858.789381s 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc1ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bad9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2bc121c20 0x7fa2bc1220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc1ca650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39896/0 (socket says 172.21.15.37:39896) 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 1 -- 172.21.15.37:0/1033817859 learned_addr learned my addr 172.21.15.37:0/1033817859 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 10 monclient(hunting): get_auth_request con 0x7fa2bc06d430 auth_method 0 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bad9d700 10 monclient(hunting): get_auth_request con 0x7fa2bc121c20 auth_method 0 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2b9d9b700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2bc06dd50 0x7fa2bc1cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2b9d9b700 10 monclient(hunting): get_auth_request con 0x7fa2bc06dd50 auth_method 0 2024-09-16T07:18:58.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2b9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2b9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2b9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2b9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2b9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2b9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 1 -- 172.21.15.37:0/1033817859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2bc06dd50 msgr2=0x7fa2bc1cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2bc06dd50 0x7fa2bc1cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 1 -- 172.21.15.37:0/1033817859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2bc121c20 msgr2=0x7fa2bc1220a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2bc121c20 0x7fa2bc1220a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 1 monclient: found mon.noname-c 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 1 -- 172.21.15.37:0/1033817859 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac009570 con 0x7fa2bc06d430 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bad9d700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2bc121c20 0x7fa2bc1220a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:58.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2ba59c700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc1ca650 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0a0f10 tx=0x7fa2ac004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 1 -- 172.21.15.37:0/1033817859 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac004510 con 0x7fa2bc06d430 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 10 monclient: dump: 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.318 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 1 -- 172.21.15.37:0/1033817859 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac01e070 con 0x7fa2bc06d430 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 5 monclient: authenticate success, global_id 4235 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient: _renew_subs 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 1 -- 172.21.15.37:0/1033817859 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc125900 con 0x7fa2bc06d430 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 1 -- 172.21.15.37:0/1033817859 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac003900 con 0x7fa2bc06d430 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2aaffd700 10 monclient: dump: 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2b8d99700 4 set_mon_vals no callback set 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient: _renew_subs 2024-09-16T07:18:58.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:58.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.300+0000 7fa2bb59e700 1 -- 172.21.15.37:0/1033817859 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc09dda0 con 0x7fa2bc06d430 2024-09-16T07:18:58.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.304+0000 7fa2aaffd700 1 -- 172.21.15.37:0/1033817859 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fa2ac003a60 con 0x7fa2bc06d430 2024-09-16T07:18:58.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.304+0000 7fa2aaffd700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa29c035630 0x7fa29c037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.304+0000 7fa2bb59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:58.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.304+0000 7fa2bb59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:58.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.304+0000 7fa2bb59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:58.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.304+0000 7fa2aaffd700 1 -- 172.21.15.37:0/1033817859 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7fa2ac048610 con 0x7fa2bc06d430 2024-09-16T07:18:58.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.304+0000 7fa2bb59e700 1 -- 172.21.15.37:0/1033817859 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc070360 con 0x7fa2bc06d430 2024-09-16T07:18:58.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.304+0000 7fa2b9d9b700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa29c035630 0x7fa29c037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.304+0000 7fa2b9d9b700 10 monclient: get_auth_request con 0x7fa29c035630 auth_method 0 2024-09-16T07:18:58.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.304+0000 7fa2b9d9b700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa29c035630 0x7fa29c037af0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc123480 tx=0x7fa2b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.312+0000 7fa2aaffd700 1 -- 172.21.15.37:0/1033817859 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa2bc070360 con 0x7fa2bc06d430 2024-09-16T07:18:58.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.312+0000 7fa2aaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:58.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.312+0000 7fa2aaffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:58.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.484+0000 7fa2bb59e700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:58.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.484+0000 7fa2bb59e700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:58.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.484+0000 7fa2bb59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:58.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.484+0000 7fa2bb59e700 1 -- 172.21.15.37:0/1033817859 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2bc070360 con 0x7fa2bc06d430 2024-09-16T07:18:58.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.484+0000 7fa2aaffd700 1 -- 172.21.15.37:0/1033817859 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+12276 (secure 0 0 0) 0x7fa2bc070360 con 0x7fa2bc06d430 2024-09-16T07:18:58.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.484+0000 7fa2aaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:18:58.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.484+0000 7fa2aaffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:58.485 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:18:58.486 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":20,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","created":"2024-09-16T07:18:36.646941+0000","modified":"2024-09-16T07:18:57.461117+0000","last_up_change":"2024-09-16T07:18:46.372944+0000","last_in_change":"2024-09-16T07:18:38.297482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T07:18:46.507362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T07:18:53.543683+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd764fc8-b5fb-40fe-a64f-ce7b8c1df525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6801","nonce":3137250735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6803","nonce":3137250735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6807","nonce":3137250735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6805","nonce":3137250735}]},"public_addr":"172.21.15.37:6801/3137250735","cluster_addr":"172.21.15.37:6803/3137250735","heartbeat_back_addr":"172.21.15.37:6807/3137250735","heartbeat_front_addr":"172.21.15.37:6805/3137250735","state":["exists","up"]},{"osd":1,"uuid":"52d577a5-08b7-452d-aa35-e1560215a09e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6809","nonce":810396637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6811","nonce":810396637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6815","nonce":810396637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6813","nonce":810396637}]},"public_addr":"172.21.15.37:6809/810396637","cluster_addr":"172.21.15.37:6811/810396637","heartbeat_back_addr":"172.21.15.37:6815/810396637","heartbeat_front_addr":"172.21.15.37:6813/810396637","state":["exists","up"]},{"osd":2,"uuid":"e3deb985-fb4e-42ee-8ed6-e6318cbca322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6817","nonce":265940208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6819","nonce":265940208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6823","nonce":265940208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6821","nonce":265940208}]},"public_addr":"172.21.15.37:6817/265940208","cluster_addr":"172.21.15.37:6819/265940208","heartbeat_back_addr":"172.21.15.37:6823/265940208","heartbeat_front_addr":"172.21.15.37:6821/265940208","state":["exists","up"]},{"osd":3,"uuid":"ab447540-8ed1-4a0c-8f6c-dfeea8af02ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6811","nonce":2588720019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6815","nonce":2588720019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6823","nonce":2588720019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6819","nonce":2588720019}]},"public_addr":"172.21.15.143:6811/2588720019","cluster_addr":"172.21.15.143:6815/2588720019","heartbeat_back_addr":"172.21.15.143:6823/2588720019","heartbeat_front_addr":"172.21.15.143:6819/2588720019","state":["exists","up"]},{"osd":4,"uuid":"be421fc0-1edf-4f20-b530-c9cf4eb20a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6810","nonce":3355421705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6814","nonce":3355421705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6822","nonce":3355421705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6818","nonce":3355421705}]},"public_addr":"172.21.15.143:6810/3355421705","cluster_addr":"172.21.15.143:6814/3355421705","heartbeat_back_addr":"172.21.15.143:6822/3355421705","heartbeat_front_addr":"172.21.15.143:6818/3355421705","state":["exists","up"]},{"osd":5,"uuid":"8c7725bd-5f56-4fa0-b455-6569e313f9fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6801","nonce":3921632375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6803","nonce":3921632375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6807","nonce":3921632375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6805","nonce":3921632375}]},"public_addr":"172.21.15.143:6801/3921632375","cluster_addr":"172.21.15.143:6803/3921632375","heartbeat_back_addr":"172.21.15.143:6807/3921632375","heartbeat_front_addr":"172.21.15.143:6805/3921632375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.492199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.565060+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.519201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.777218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.842229+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.307398+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:18:58.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 -- 172.21.15.37:0/1033817859 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa29c035630 msgr2=0x7fa29c037af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa29c035630 0x7fa29c037af0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc123480 tx=0x7fa2b0009040 comp rx=0 tx=0).stop 2024-09-16T07:18:58.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 10 monclient: shutdown 2024-09-16T07:18:58.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 -- 172.21.15.37:0/1033817859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 msgr2=0x7fa2bc1ca650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc1ca650 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0a0f10 tx=0x7fa2ac004770 comp rx=0 tx=0).stop 2024-09-16T07:18:58.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 -- 172.21.15.37:0/1033817859 shutdown_connections 2024-09-16T07:18:58.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2bc121c20 0x7fa2bc1220a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2bc06dd50 0x7fa2bc1cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa29c035630 0x7fa29c037af0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 --2- 172.21.15.37:0/1033817859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2bc06d430 0x7fa2bc1ca650 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 -- 172.21.15.37:0/1033817859 >> 172.21.15.37:0/1033817859 conn(0x7fa2bc0a14c0 msgr2=0x7fa2bc06a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 -- 172.21.15.37:0/1033817859 shutdown_connections 2024-09-16T07:18:58.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.488+0000 7fa2bb59e700 1 -- 172.21.15.37:0/1033817859 wait complete. 2024-09-16T07:18:58.500 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T07:18:58.500 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T07:18:58.501 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T07:18:58.501 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T07:18:58.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T07:18:58.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T07:18:58.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.616+0000 7fde79d9e700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.616+0000 7fde79d9e700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.616+0000 7fde79d9e700 10 monclient: monmap: 2024-09-16T07:18:58.617 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.617 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.617 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.621791+0000 2024-09-16T07:18:58.618 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.621791+0000 2024-09-16T07:18:58.618 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.618 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.618 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.618 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.618 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.618 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.618 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.618 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.616+0000 7fde79d9e700 1 Processor -- start 2024-09-16T07:18:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.616+0000 7fde79d9e700 1 -- start start 2024-09-16T07:18:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.616+0000 7fde79d9e700 10 monclient: init 2024-09-16T07:18:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde740a1450 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde740a1f40 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde74139b60 0x7fde7413bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient(hunting): picked mon.noname-c con 0x7fde74139b60 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde74071c70 con 0x7fde740a1450 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde740713b0 con 0x7fde740a1f40 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde74070a40 con 0x7fde74139b60 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient(hunting): authenticate will time out at 859.108552s 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde78d9c700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fde740a1450 0x7fde740a1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36588/0 (socket says 172.21.15.37:36588) 2024-09-16T07:18:58.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde78d9c700 1 -- 172.21.15.37:0/2837520873 learned_addr learned my addr 172.21.15.37:0/2837520873 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:18:58.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 1 --2- 172.21.15.37:0/2837520873 >> v2:172.21.15.143:3300/0 conn(0x7fde74139b60 0x7fde7413bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 10 monclient(hunting): get_auth_request con 0x7fde74139b60 auth_method 0 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 1 -- 172.21.15.37:0/2837520873 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790280355 0 0) 0x7fde74071c70 con 0x7fde740a1450 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient(hunting): _negotiate 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient(hunting): my global_id is 4248 2024-09-16T07:18:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 1 -- 172.21.15.37:0/2837520873 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde5c0035d0 con 0x7fde740a1450 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 1 -- 172.21.15.37:0/2837520873 >> v1:172.21.15.143:6789/0 conn(0x7fde740a1f40 legacy=0x7fde74138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 1 -- 172.21.15.37:0/2837520873 >> v1:172.21.15.37:6789/0 conn(0x7fde740a1450 legacy=0x7fde740a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 1 monclient: found mon.noname-c 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 1 -- 172.21.15.37:0/2837520873 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7406ee60 con 0x7fde74139b60 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde7959d700 1 --2- 172.21.15.37:0/2837520873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde74139b60 0x7fde7413bf50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fde64009850 tx=0x7fde64009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49286868531cb7b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 1 -- 172.21.15.37:0/2837520873 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64011830 con 0x7fde74139b60 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient: dump: 2024-09-16T07:18:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 5 monclient: authenticate success, global_id 4270 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 1 -- 172.21.15.37:0/2837520873 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde64011990 con 0x7fde74139b60 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 1 -- 172.21.15.37:0/2837520873 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64011d00 con 0x7fde74139b60 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 4 set_mon_vals no callback set 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde737fe700 10 monclient: dump: 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient: shutdown 2024-09-16T07:18:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 -- 172.21.15.37:0/2837520873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde74139b60 msgr2=0x7fde7413bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 --2- 172.21.15.37:0/2837520873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde74139b60 0x7fde7413bf50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fde64009850 tx=0x7fde64009b60 comp rx=0 tx=0).stop 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 -- 172.21.15.37:0/2837520873 shutdown_connections 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 --2- 172.21.15.37:0/2837520873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde74139b60 0x7fde7413bf50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 -- 172.21.15.37:0/2837520873 >> 172.21.15.37:0/2837520873 conn(0x7fde740a2540 msgr2=0x7fde740a2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 -- 172.21.15.37:0/2837520873 shutdown_connections 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 1 -- 172.21.15.37:0/2837520873 wait complete. 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.620+0000 7fde79d9e700 10 monclient: monmap: 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:58.629 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 1 Processor -- start 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 1 -- start start 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient: init 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde740a1450 0x7fde7411c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient(hunting): picked mon.noname-c con 0x7fde740a1450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde740a1f40 0x7fde7411c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient(hunting): picked mon.noname-b con 0x7fde740a1f40 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde74139b60 0x7fde7411cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient(hunting): picked mon.noname-a con 0x7fde74139b60 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7411e800 con 0x7fde74139b60 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7411ea50 con 0x7fde740a1f40 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde740726f0 con 0x7fde740a1450 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient(hunting): authenticate will time out at 859.113281s 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde78d9c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde740a1450 0x7fde7411c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde78d9c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde740a1450 0x7fde7411c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39904/0 (socket says 172.21.15.37:39904) 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde78d9c700 1 -- 172.21.15.37:0/3030170506 learned_addr learned my addr 172.21.15.37:0/3030170506 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde78d9c700 10 monclient(hunting): get_auth_request con 0x7fde740a1450 auth_method 0 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde78d9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde78d9c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde78d9c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde74139b60 0x7fde7411cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 10 monclient(hunting): get_auth_request con 0x7fde74139b60 auth_method 0 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde78d9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde78d9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde78d9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-09-16T07:18:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 1 -- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde740a1450 msgr2=0x7fde7411c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde740a1450 0x7fde7411c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 1 -- 172.21.15.37:0/3030170506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde740a1f40 msgr2=0x7fde7411c5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:18:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde740a1f40 0x7fde7411c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 1 -- 172.21.15.37:0/3030170506 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde640094b0 con 0x7fde74139b60 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde7959d700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde74139b60 0x7fde7411cb10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fde64016000 tx=0x7fde64004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 1 -- 172.21.15.37:0/3030170506 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde640045d0 con 0x7fde74139b60 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 10 monclient: dump: 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 5 monclient: authenticate success, global_id 4251 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient: _renew_subs 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde741391b0 con 0x7fde74139b60 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient: _renew_subs 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7409f2a0 con 0x7fde74139b60 2024-09-16T07:18:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 1 -- 172.21.15.37:0/3030170506 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde640036c0 con 0x7fde74139b60 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 1 -- 172.21.15.37:0/3030170506 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64003a70 con 0x7fde74139b60 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde72ffd700 4 set_mon_vals no callback set 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.624+0000 7fde717fa700 10 monclient: dump: 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde717fa700 1 -- 172.21.15.37:0/3030170506 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fde64021070 con 0x7fde74139b60 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde717fa700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fde60035600 0x7fde60037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde717fa700 1 -- 172.21.15.37:0/3030170506 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7fde64048fd0 con 0x7fde74139b60 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde78d9c700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fde60035600 0x7fde60037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde78d9c700 10 monclient: get_auth_request con 0x7fde60035600 auth_method 0 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde79d9e700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x7fde540015e0 0x7fde54003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 --> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde54006b40 con 0x7fde540015e0 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde78d9c700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fde60035600 0x7fde60037ac0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde6c001560 tx=0x7fde6c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde73fff700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x7fde540015e0 0x7fde54003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde73fff700 10 monclient: get_auth_request con 0x7fde540015e0 auth_method 0 2024-09-16T07:18:58.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.628+0000 7fde73fff700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x7fde540015e0 0x7fde54003aa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.632+0000 7fde717fa700 1 -- 172.21.15.37:0/3030170506 <== osd.1 v2:172.21.15.37:6808/810396637 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fde54006b40 con 0x7fde540015e0 2024-09-16T07:18:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.632+0000 7fc716cfa700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.632+0000 7fc716cfa700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.632+0000 7fc716cfa700 10 monclient: monmap: 2024-09-16T07:18:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.635532+0000 2024-09-16T07:18:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.635532+0000 2024-09-16T07:18:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.632+0000 7fc716cfa700 1 Processor -- start 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.632+0000 7fc716cfa700 1 -- start start 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.632+0000 7fc716cfa700 10 monclient: init 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T07:18:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc710071170 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc71009fa00 0x7fc71013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient(hunting): picked mon.noname-a con 0x7fc71009fa00 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc71013abf0 0x7fc71013cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient(hunting): picked mon.noname-c con 0x7fc71013abf0 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71009d6b0 con 0x7fc710071170 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc710099de0 con 0x7fc71009fa00 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc71009a0c0 con 0x7fc71013abf0 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient(hunting): authenticate will time out at 859.124627s 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc714a96700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc710071170 0x7fc710071550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36594/0 (socket says 172.21.15.37:36594) 2024-09-16T07:18:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc71009fa00 0x7fc71013a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc714a96700 1 -- 172.21.15.37:0/2817688085 learned_addr learned my addr 172.21.15.37:0/2817688085 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 10 monclient(hunting): get_auth_request con 0x7fc71009fa00 auth_method 0 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc715297700 1 --2- 172.21.15.37:0/2817688085 >> v2:172.21.15.143:3300/0 conn(0x7fc71013abf0 0x7fc71013cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc715297700 10 monclient(hunting): get_auth_request con 0x7fc71013abf0 auth_method 0 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc715297700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc715297700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc715297700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 10 monclient(hunting): handle_auth_done global_id 4254 payload 274 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 1 -- 172.21.15.37:0/2817688085 >> v2:172.21.15.143:3300/0 conn(0x7fc71013abf0 msgr2=0x7fc71013cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 1 --2- 172.21.15.37:0/2817688085 >> v2:172.21.15.143:3300/0 conn(0x7fc71013abf0 0x7fc71013cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 1 -- 172.21.15.37:0/2817688085 >> v1:172.21.15.37:6789/0 conn(0x7fc710071170 legacy=0x7fc710071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 1 -- 172.21.15.37:0/2817688085 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7100a1240 con 0x7fc71009fa00 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70ffff700 1 --2- 172.21.15.37:0/2817688085 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc71009fa00 0x7fc71013a6b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc71009d470 tx=0x7fc704009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17923b4c249da372 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 1 -- 172.21.15.37:0/2817688085 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704011830 con 0x7fc71009fa00 2024-09-16T07:18:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 10 monclient: dump: 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.648 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 5 monclient: authenticate success, global_id 4254 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 1 -- 172.21.15.37:0/2817688085 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704011990 con 0x7fc71009fa00 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 4 set_mon_vals no callback set 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 1 -- 172.21.15.37:0/2817688085 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704011d40 con 0x7fc71009fa00 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc70f7fe700 10 monclient: dump: 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 10 monclient: shutdown 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 1 -- 172.21.15.37:0/2817688085 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc71009fa00 msgr2=0x7fc71013a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.636+0000 7fc716cfa700 1 --2- 172.21.15.37:0/2817688085 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc71009fa00 0x7fc71013a6b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc71009d470 tx=0x7fc704009b30 comp rx=0 tx=0).stop 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 -- 172.21.15.37:0/2817688085 shutdown_connections 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 --2- 172.21.15.37:0/2817688085 >> v2:172.21.15.143:3300/0 conn(0x7fc71013abf0 0x7fc71013cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 --2- 172.21.15.37:0/2817688085 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc71009fa00 0x7fc71013a6b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 -- 172.21.15.37:0/2817688085 >> 172.21.15.37:0/2817688085 conn(0x7fc7100a04b0 msgr2=0x7fc7100a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 -- 172.21.15.37:0/2817688085 shutdown_connections 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 -- 172.21.15.37:0/2817688085 wait complete. 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient: monmap: 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 Processor -- start 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 -- start start 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient: init 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc710071170 0x7fc7101ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient(hunting): picked mon.noname-a con 0x7fc710071170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7ff08a806700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7ff08a806700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7ff08a806700 10 monclient: monmap: 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.644951+0000 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.644951+0000 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.653 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.654 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.654 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71009fa00 0x7fc7101354a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient(hunting): picked mon.noname-b con 0x7fc71009fa00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc71013abf0 0x7fc71011bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient(hunting): picked mon.noname-c con 0x7fc71013abf0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71011f5b0 con 0x7fc710071170 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc710072660 con 0x7fc71009fa00 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7101cb430 con 0x7fc71013abf0 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient(hunting): authenticate will time out at 859.130429s 2024-09-16T07:18:58.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc710071170 0x7fc7101ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc715297700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc71013abf0 0x7fc71011bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc710071170 0x7fc7101ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36410/0 (socket says 172.21.15.37:36410) 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 1 -- 172.21.15.37:0/2147210125 learned_addr learned my addr 172.21.15.37:0/2147210125 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 10 monclient(hunting): get_auth_request con 0x7fc710071170 auth_method 0 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc715297700 10 monclient(hunting): get_auth_request con 0x7fc71013abf0 auth_method 0 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc715297700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc715297700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc715297700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc715297700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc715297700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc715297700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70ffff700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71009fa00 0x7fc7101354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70ffff700 10 monclient(hunting): get_auth_request con 0x7fc71009fa00 auth_method 0 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 1 -- 172.21.15.37:0/2147210125 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc71013abf0 msgr2=0x7fc71011bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc71013abf0 0x7fc71011bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 1 -- 172.21.15.37:0/2147210125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71009fa00 msgr2=0x7fc7101354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71009fa00 0x7fc7101354a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 1 -- 172.21.15.37:0/2147210125 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040094b0 con 0x7fc710071170 2024-09-16T07:18:58.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70ffff700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc71009fa00 0x7fc7101354a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:18:58.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7faee027d700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7faee027d700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7faee027d700 10 monclient: monmap: 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.646138+0000 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.646138+0000 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.659 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7ff08a806700 1 Processor -- start 2024-09-16T07:18:58.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc714a96700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc710071170 0x7fc7101ca4a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc70800caf0 tx=0x7fc70800cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7ff08a806700 1 -- start start 2024-09-16T07:18:58.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7ff08a806700 10 monclient: init 2024-09-16T07:18:58.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 1 -- 172.21.15.37:0/2147210125 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc708007480 con 0x7fc710071170 2024-09-16T07:18:58.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:58.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7faee027d700 1 Processor -- start 2024-09-16T07:18:58.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 10 monclient: dump: 2024-09-16T07:18:58.662 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.662 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.662 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.662 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.662 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.662 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.662 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.662 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 1 -- 172.21.15.37:0/2147210125 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7080075e0 con 0x7fc710071170 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 1 -- 172.21.15.37:0/2147210125 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc708007970 con 0x7fc710071170 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70d7fa700 10 monclient: dump: 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.663 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 5 monclient: authenticate success, global_id 4260 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient: _renew_subs 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71011daf0 con 0x7fc710071170 2024-09-16T07:18:58.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7faee027d700 1 -- start start 2024-09-16T07:18:58.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7faee027d700 10 monclient: init 2024-09-16T07:18:58.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7faed809d8d0 0x7faed80999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient(hunting): picked mon.noname-a con 0x7faed809d8d0 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faed809dcb0 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed81371b0 0x7faed81395a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient(hunting): picked mon.noname-b con 0x7faed81371b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed809bb40 con 0x7faed809dcb0 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed809be20 con 0x7faed809d8d0 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed809c100 con 0x7faed81371b0 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient(hunting): authenticate will time out at 859.132519s 2024-09-16T07:18:58.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T07:18:58.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0840a16f0 0x7ff08409d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 10 monclient(hunting): picked mon.noname-b con 0x7ff0840a16f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff08409d9d0 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:58.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0841317f0 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:58.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08409a720 con 0x7ff0841317f0 2024-09-16T07:18:58.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08409b9a0 con 0x7ff08409d9d0 2024-09-16T07:18:58.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7faed809d8d0 0x7faed80999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7faed809d8d0 0x7faed80999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36412/0 (socket says 172.21.15.37:36412) 2024-09-16T07:18:58.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 1 -- 172.21.15.37:0/951299504 learned_addr learned my addr 172.21.15.37:0/951299504 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 10 monclient(hunting): get_auth_request con 0x7faed809d8d0 auth_method 0 2024-09-16T07:18:58.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc70effd700 4 set_mon_vals no callback set 2024-09-16T07:18:58.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient: _renew_subs 2024-09-16T07:18:58.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.640+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71011e4f0 con 0x7fc710071170 2024-09-16T07:18:58.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08409a470 con 0x7ff0840a16f0 2024-09-16T07:18:58.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 10 monclient(hunting): authenticate will time out at 859.132478s 2024-09-16T07:18:58.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede81a700 1 --2- 172.21.15.37:0/951299504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed81371b0 0x7faed81395a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2024-09-16T07:18:58.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 1 -- 172.21.15.37:0/951299504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed81371b0 msgr2=0x7faed81395a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0840a16f0 0x7ff08409d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0840a16f0 0x7ff08409d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39928/0 (socket says 172.21.15.37:39928) 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 1 -- 172.21.15.37:0/3539148475 learned_addr learned my addr 172.21.15.37:0/3539148475 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 10 monclient(hunting): get_auth_request con 0x7ff0840a16f0 auth_method 0 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-09-16T07:18:58.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 1 -- 172.21.15.37:0/3539148475 >> v1:172.21.15.143:6789/0 conn(0x7ff08409d9d0 legacy=0x7ff0841310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 1 -- 172.21.15.37:0/3539148475 >> v1:172.21.15.37:6789/0 conn(0x7ff0841317f0 legacy=0x7ff0841323e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 1 monclient: found mon.noname-b 2024-09-16T07:18:58.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:18:58.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 1 -- 172.21.15.37:0/3539148475 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff084131c20 con 0x7ff0840a16f0 2024-09-16T07:18:58.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff089804700 1 --2- 172.21.15.37:0/3539148475 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0840a16f0 0x7ff08409d490 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff080009940 tx=0x7ff080009c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6de1a034b834e5a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 1 --2- 172.21.15.37:0/951299504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed81371b0 0x7faed81395a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 1 -- 172.21.15.37:0/951299504 >> v1:172.21.15.143:6789/0 conn(0x7faed809dcb0 legacy=0x7faed8099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 1 -- 172.21.15.37:0/3539148475 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080004230 con 0x7ff0840a16f0 2024-09-16T07:18:58.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:58.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 1 -- 172.21.15.37:0/951299504 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed809b040 con 0x7faed809d8d0 2024-09-16T07:18:58.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 10 monclient: dump: 2024-09-16T07:18:58.678 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.678 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 1 -- 172.21.15.37:0/3539148475 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff080009e90 con 0x7ff0840a16f0 2024-09-16T07:18:58.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 1 -- 172.21.15.37:0/3539148475 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800045b0 con 0x7ff0840a16f0 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff088802700 10 monclient: dump: 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.681 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 5 monclient: authenticate success, global_id 4244 2024-09-16T07:18:58.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:58.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 4 set_mon_vals no callback set 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede019700 1 --2- 172.21.15.37:0/951299504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faed809d8d0 0x7faed80999e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faed000d3a0 tx=0x7faed000d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73765b3cf114b28 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede81a700 10 monclient: get_auth_request con 0x7faed81371b0 auth_method 0 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faede81a700 1 --2- 172.21.15.37:0/951299504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed81371b0 0x7faed81395a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 1 -- 172.21.15.37:0/951299504 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0010070 con 0x7faed809d8d0 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 10 monclient: dump: 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.682 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 5 monclient: authenticate success, global_id 4266 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 1 -- 172.21.15.37:0/951299504 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed000dd00 con 0x7faed809d8d0 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 4 set_mon_vals no callback set 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 1 -- 172.21.15.37:0/951299504 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0007d70 con 0x7faed809d8d0 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faedd017700 10 monclient: dump: 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.684 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.685 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.685 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7fc70d7fa700 1 -- 172.21.15.37:0/2147210125 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fc70801b3b0 con 0x7fc710071170 2024-09-16T07:18:58.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 10 monclient: shutdown 2024-09-16T07:18:58.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 -- 172.21.15.37:0/3539148475 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0840a16f0 msgr2=0x7ff08409d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 --2- 172.21.15.37:0/3539148475 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0840a16f0 0x7ff08409d490 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff080009940 tx=0x7ff080009c50 comp rx=0 tx=0).stop 2024-09-16T07:18:58.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 10 monclient: shutdown 2024-09-16T07:18:58.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 1 -- 172.21.15.37:0/951299504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faed809d8d0 msgr2=0x7faed80999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7faee027d700 1 --2- 172.21.15.37:0/951299504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faed809d8d0 0x7faed80999e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faed000d3a0 tx=0x7faed000d6b0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7fc70d7fa700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc6f4035600 0x7fc6f4037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7fc70d7fa700 1 -- 172.21.15.37:0/2147210125 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7fc7080474b0 con 0x7fc710071170 2024-09-16T07:18:58.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 -- 172.21.15.37:0/3539148475 shutdown_connections 2024-09-16T07:18:58.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 --2- 172.21.15.37:0/3539148475 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0840a16f0 0x7ff08409d490 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 -- 172.21.15.37:0/3539148475 >> 172.21.15.37:0/3539148475 conn(0x7ff0840a3c60 msgr2=0x7ff0840a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7fc716cfa700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x7fc7101359e0 0x7fc71011c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 -- 172.21.15.37:0/3539148475 shutdown_connections 2024-09-16T07:18:58.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7fc70ffff700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc6f4035600 0x7fc6f4037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 --> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc71011feb0 con 0x7fc7101359e0 2024-09-16T07:18:58.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7ff08a806700 1 -- 172.21.15.37:0/3539148475 wait complete. 2024-09-16T07:18:58.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.644+0000 7fc70ffff700 10 monclient: get_auth_request con 0x7fc6f4035600 auth_method 0 2024-09-16T07:18:58.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7fc715297700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x7fc7101359e0 0x7fc71011c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7fc715297700 10 monclient: get_auth_request con 0x7fc7101359e0 auth_method 0 2024-09-16T07:18:58.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 -- 172.21.15.37:0/951299504 shutdown_connections 2024-09-16T07:18:58.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 --2- 172.21.15.37:0/951299504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed81371b0 0x7faed81395a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 --2- 172.21.15.37:0/951299504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faed809d8d0 0x7faed80999e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 -- 172.21.15.37:0/951299504 >> 172.21.15.37:0/951299504 conn(0x7faed80a04b0 msgr2=0x7faed80a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 -- 172.21.15.37:0/951299504 shutdown_connections 2024-09-16T07:18:58.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 -- 172.21.15.37:0/951299504 wait complete. 2024-09-16T07:18:58.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7fc715297700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x7fc7101359e0 0x7fc71011c420 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7fc70ffff700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc6f4035600 0x7fc6f4037ac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc7040090b0 tx=0x7fc704008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient: monmap: 2024-09-16T07:18:58.691 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.691 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:58.691 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:58.691 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:58.691 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.691 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:58.691 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:58.692 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:58.692 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:58.692 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient: monmap: 2024-09-16T07:18:58.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:58.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:58.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:58.693 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.693 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:58.693 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:58.693 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:58.693 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:58.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 Processor -- start 2024-09-16T07:18:58.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 1 Processor -- start 2024-09-16T07:18:58.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 -- start start 2024-09-16T07:18:58.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient: init 2024-09-16T07:18:58.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 1 -- start start 2024-09-16T07:18:58.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient: init 2024-09-16T07:18:58.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7fc70d7fa700 1 -- 172.21.15.37:0/2147210125 <== osd.0 v2:172.21.15.37:6800/3137250735 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fc71011feb0 con 0x7fc7101359e0 2024-09-16T07:18:58.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:18:58.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed809d8d0 0x7faed81ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient(hunting): picked mon.noname-c con 0x7faed809d8d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faed81371b0 0x7faed8136480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient(hunting): picked mon.noname-b con 0x7faed81371b0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:58.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faed81369c0 0x7faed8122f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient(hunting): picked mon.noname-a con 0x7faed81369c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed81cab40 con 0x7faed81369c0 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed8136e50 con 0x7faed81371b0 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed8126b90 con 0x7faed809d8d0 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient(hunting): authenticate will time out at 859.137937s 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faed81369c0 0x7faed8122f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faed81369c0 0x7faed8122f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36418/0 (socket says 172.21.15.37:36418) 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 1 -- 172.21.15.37:0/1005645073 learned_addr learned my addr 172.21.15.37:0/1005645073 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 10 monclient(hunting): get_auth_request con 0x7faed81369c0 auth_method 0 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede019700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed809d8d0 0x7faed81ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede019700 10 monclient(hunting): get_auth_request con 0x7faed809d8d0 auth_method 0 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede019700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede019700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede019700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede019700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede019700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede019700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2024-09-16T07:18:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 1 -- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed809d8d0 msgr2=0x7faed81ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed809d8d0 0x7faed81ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 1 -- 172.21.15.37:0/1005645073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faed81371b0 msgr2=0x7faed8136480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T07:18:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faed81371b0 0x7faed8136480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 1 -- 172.21.15.37:0/1005645073 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed000d050 con 0x7faed81369c0 2024-09-16T07:18:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede019700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faed809d8d0 0x7faed81ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:58.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:18:58.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff08409d9d0 0x7ff0841309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient(hunting): picked mon.noname-a con 0x7ff08409d9d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:58.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0840a16f0 0x7ff08412b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faede81a700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faed81369c0 0x7faed8122f10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faec800c7f0 tx=0x7faec800cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient(hunting): picked mon.noname-b con 0x7ff0840a16f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0841317f0 0x7ff08412bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient(hunting): picked mon.noname-c con 0x7ff0841317f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08409e680 con 0x7ff08409d9d0 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0840724c0 con 0x7ff0840a16f0 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08412fee0 con 0x7ff0841317f0 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a806700 10 monclient(hunting): authenticate will time out at 859.138151s 2024-09-16T07:18:58.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff08409d9d0 0x7ff0841309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff08409d9d0 0x7ff0841309f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36420/0 (socket says 172.21.15.37:36420) 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 1 -- 172.21.15.37:0/1269860767 learned_addr learned my addr 172.21.15.37:0/1269860767 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 10 monclient(hunting): get_auth_request con 0x7ff08409d9d0 auth_method 0 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a005700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0841317f0 0x7ff08412bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a005700 10 monclient(hunting): get_auth_request con 0x7ff0841317f0 auth_method 0 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a005700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a005700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a005700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089003700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0840a16f0 0x7ff08412b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089003700 10 monclient(hunting): get_auth_request con 0x7ff0840a16f0 auth_method 0 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089003700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089003700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089003700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a005700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a005700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff08a005700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089003700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089003700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089003700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 1 -- 172.21.15.37:0/1269860767 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0841317f0 msgr2=0x7ff08412bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0841317f0 0x7ff08412bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 1 -- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0840a16f0 msgr2=0x7ff08412b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0840a16f0 0x7ff08412b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 1 -- 172.21.15.37:0/1269860767 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff080009570 con 0x7ff08409d9d0 2024-09-16T07:18:58.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7ff089804700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff08409d9d0 0x7ff0841309f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff080009150 tx=0x7ff080004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 1 -- 172.21.15.37:0/1005645073 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec8004220 con 0x7faed81369c0 2024-09-16T07:18:58.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:58.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 10 monclient: dump: 2024-09-16T07:18:58.706 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 1 -- 172.21.15.37:0/1005645073 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec80043a0 con 0x7faed81369c0 2024-09-16T07:18:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 5 monclient: authenticate success, global_id 4269 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient: _renew_subs 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 1 -- 172.21.15.37:0/1005645073 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec80036c0 con 0x7faed81369c0 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed81256e0 con 0x7faed81369c0 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faeceffd700 10 monclient: dump: 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.708 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.709 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.709 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.709 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.709 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.709 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient: _renew_subs 2024-09-16T07:18:58.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.648+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed81268b0 con 0x7faed81369c0 2024-09-16T07:18:58.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 1 -- 172.21.15.37:0/1269860767 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08000bd20 con 0x7ff08409d9d0 2024-09-16T07:18:58.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:58.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 10 monclient: dump: 2024-09-16T07:18:58.710 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 1 -- 172.21.15.37:0/1269860767 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08001f070 con 0x7ff08409d9d0 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 1 -- 172.21.15.37:0/1269860767 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080003900 con 0x7ff08409d9d0 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 10 monclient: dump: 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.712 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff08a806700 5 monclient: authenticate success, global_id 4272 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff08a806700 10 monclient: _renew_subs 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff08a806700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff08a806700 1 -- 172.21.15.37:0/1269860767 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08412ea60 con 0x7ff08409d9d0 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff08a806700 10 monclient: _renew_subs 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff08a806700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff08a806700 1 -- 172.21.15.37:0/1269860767 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08412dd90 con 0x7ff08409d9d0 2024-09-16T07:18:58.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07bfff700 4 set_mon_vals no callback set 2024-09-16T07:18:58.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faeceffd700 1 -- 172.21.15.37:0/1005645073 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7faec8003840 con 0x7faed81369c0 2024-09-16T07:18:58.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faeceffd700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faec4035760 0x7faec4037c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faecffff700 4 set_mon_vals no callback set 2024-09-16T07:18:58.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faeceffd700 1 -- 172.21.15.37:0/1005645073 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7faec80477b0 con 0x7faed81369c0 2024-09-16T07:18:58.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 1 -- 172.21.15.37:0/1269860767 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7ff080021030 con 0x7ff08409d9d0 2024-09-16T07:18:58.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faede019700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faec4035760 0x7faec4037c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faede019700 10 monclient: get_auth_request con 0x7faec4035760 auth_method 0 2024-09-16T07:18:58.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff070035760 0x7ff070037c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff07a7fc700 1 -- 172.21.15.37:0/1269860767 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7ff080049180 con 0x7ff08409d9d0 2024-09-16T07:18:58.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff089003700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff070035760 0x7ff070037c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff089003700 10 monclient: get_auth_request con 0x7ff070035760 auth_method 0 2024-09-16T07:18:58.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faee027d700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] conn(0x7faebc0015e0 0x7faebc003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 --> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faebc006b40 con 0x7faebc0015e0 2024-09-16T07:18:58.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faede019700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faec4035760 0x7faec4037c20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faed000d6f0 tx=0x7faed0012000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faedd818700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] conn(0x7faebc0015e0 0x7faebc003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faedd818700 10 monclient: get_auth_request con 0x7faebc0015e0 auth_method 0 2024-09-16T07:18:58.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7ff089003700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff070035760 0x7ff070037c20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff084132100 tx=0x7ff074007ee0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.652+0000 7faedd818700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] conn(0x7faebc0015e0 0x7faebc003aa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7faeceffd700 1 -- 172.21.15.37:0/1005645073 <== osd.2 v2:172.21.15.37:6816/265940208 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7faebc006b40 con 0x7faebc0015e0 2024-09-16T07:18:58.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7ff06ffff700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] conn(0x7ff0640015e0 0x7ff064003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7ff06ffff700 1 -- 172.21.15.37:0/1269860767 --> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff064006b40 con 0x7ff0640015e0 2024-09-16T07:18:58.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7ff08a005700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] conn(0x7ff0640015e0 0x7ff064003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7ff08a005700 10 monclient: get_auth_request con 0x7ff0640015e0 auth_method 0 2024-09-16T07:18:58.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7ff08a005700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] conn(0x7ff0640015e0 0x7ff064003aa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 --> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fde54005c20 con 0x7fde540015e0 2024-09-16T07:18:58.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7f121c15c700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7f121c15c700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7f121c15c700 10 monclient: monmap: 2024-09-16T07:18:58.719 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.719 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.719 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.662502+0000 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.662502+0000 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7f121c15c700 1 Processor -- start 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7f121c15c700 1 -- start start 2024-09-16T07:18:58.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7f121c15c700 10 monclient: init 2024-09-16T07:18:58.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde717fa700 1 -- 172.21.15.37:0/3030170506 <== osd.1 v2:172.21.15.37:6808/810396637 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fde54005c20 con 0x7fde540015e0 2024-09-16T07:18:58.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7ff07a7fc700 1 -- 172.21.15.37:0/1269860767 <== osd.4 v2:172.21.15.143:6808/3355421705 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7ff064006b40 con 0x7ff0640015e0 2024-09-16T07:18:58.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f121409cd40 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f12141490d0 0x7f121414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 10 monclient(hunting): picked mon.noname-a con 0x7f12141490d0 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f121414ba00 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1214071d10 con 0x7f121409cd40 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12140a0100 con 0x7f121414ba00 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f121409a010 con 0x7f12141490d0 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 10 monclient(hunting): authenticate will time out at 859.148598s 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f12141490d0 0x7f121414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f12141490d0 0x7f121414b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36436/0 (socket says 172.21.15.37:36436) 2024-09-16T07:18:58.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 1 -- 172.21.15.37:0/3044728284 learned_addr learned my addr 172.21.15.37:0/3044728284 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 10 monclient(hunting): get_auth_request con 0x7f12141490d0 auth_method 0 2024-09-16T07:18:58.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7f3934c46700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7f3934c46700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.656+0000 7f3934c46700 10 monclient: monmap: 2024-09-16T07:18:58.724 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.724 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.724 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.663288+0000 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.663288+0000 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 1 Processor -- start 2024-09-16T07:18:58.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x7fde540015e0 msgr2=0x7fde54003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x7fde540015e0 0x7fde54003aa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fde60035600 msgr2=0x7fde60037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fde60035600 0x7fde60037ac0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde6c001560 tx=0x7fde6c009040 comp rx=0 tx=0).stop 2024-09-16T07:18:58.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-09-16T07:18:58.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 1 -- start start 2024-09-16T07:18:58.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient: init 2024-09-16T07:18:58.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 10 monclient: shutdown 2024-09-16T07:18:58.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde74139b60 msgr2=0x7fde7411cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 1 --2- 172.21.15.37:0/3030170506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde74139b60 0x7fde7411cb10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fde64016000 tx=0x7fde64004770 comp rx=0 tx=0).stop 2024-09-16T07:18:58.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde7959d700 1 -- 172.21.15.37:0/3030170506 reap_dead start 2024-09-16T07:18:58.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 1 -- 172.21.15.37:0/3044728284 >> v1:172.21.15.143:6789/0 conn(0x7f121414ba00 legacy=0x7f121414ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 1 -- 172.21.15.37:0/3044728284 >> v1:172.21.15.37:6789/0 conn(0x7f121409cd40 legacy=0x7f121409f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 1 -- 172.21.15.37:0/3044728284 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121409d600 con 0x7f12141490d0 2024-09-16T07:18:58.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 shutdown_connections 2024-09-16T07:18:58.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 >> 172.21.15.37:0/3030170506 conn(0x7fde740a2540 msgr2=0x7fde740a4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f12196f7700 1 --2- 172.21.15.37:0/3044728284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f12141490d0 0x7f121414b4c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1214067530 tx=0x7f1210009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f076ab463ed4d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 shutdown_connections 2024-09-16T07:18:58.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T07:18:58.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 0x7f3930139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 1 -- 172.21.15.37:0/3044728284 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1210004230 con 0x7f12141490d0 2024-09-16T07:18:58.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:58.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 10 monclient: dump: 2024-09-16T07:18:58.731 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.731 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 1 -- 172.21.15.37:0/3044728284 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1210004390 con 0x7f12141490d0 2024-09-16T07:18:58.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 5 monclient: authenticate success, global_id 4275 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 1 -- 172.21.15.37:0/3044728284 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12100036c0 con 0x7f12141490d0 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f1218ef6700 10 monclient: dump: 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.733 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.734 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.734 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient(hunting): picked mon.noname-b con 0x7f3930139b60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f393013a480 0x7f39300a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient(hunting): picked mon.noname-c con 0x7f393013a480 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:58.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f39300a2040 0x7f39300a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient(hunting): picked mon.noname-a con 0x7f39300a2040 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:58.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3930099b90 con 0x7f39300a2040 2024-09-16T07:18:58.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393009b540 con 0x7f393013a480 2024-09-16T07:18:58.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393009ddf0 con 0x7f3930139b60 2024-09-16T07:18:58.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient(hunting): authenticate will time out at 859.149912s 2024-09-16T07:18:58.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:58.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 4 set_mon_vals no callback set 2024-09-16T07:18:58.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 0x7f3930139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 0x7f3930139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39964/0 (socket says 172.21.15.37:39964) 2024-09-16T07:18:58.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 1 -- 172.21.15.37:0/3417899757 learned_addr learned my addr 172.21.15.37:0/3417899757 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 10 monclient(hunting): get_auth_request con 0x7f3930139b60 auth_method 0 2024-09-16T07:18:58.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7fde79d9e700 1 -- 172.21.15.37:0/3030170506 wait complete. 2024-09-16T07:18:58.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392effd700 1 --2- 172.21.15.37:0/3417899757 >> v2:172.21.15.143:3300/0 conn(0x7f393013a480 0x7f39300a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392ffff700 1 --2- 172.21.15.37:0/3417899757 >> v2:172.21.15.37:3300/0 conn(0x7f39300a2040 0x7f39300a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392effd700 10 monclient(hunting): get_auth_request con 0x7f393013a480 auth_method 0 2024-09-16T07:18:58.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 1 -- 172.21.15.37:0/3417899757 >> v2:172.21.15.143:3300/0 conn(0x7f393013a480 msgr2=0x7f39300a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 1 --2- 172.21.15.37:0/3417899757 >> v2:172.21.15.143:3300/0 conn(0x7f393013a480 0x7f39300a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 1 -- 172.21.15.37:0/3417899757 >> v2:172.21.15.37:3300/0 conn(0x7f39300a2040 msgr2=0x7f39300a24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 1 --2- 172.21.15.37:0/3417899757 >> v2:172.21.15.37:3300/0 conn(0x7f39300a2040 0x7f39300a24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 1 monclient: found mon.noname-b 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 1 -- 172.21.15.37:0/3417899757 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393009b260 con 0x7f3930139b60 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392ffff700 10 monclient: get_auth_request con 0x7f39300a2040 auth_method 0 2024-09-16T07:18:58.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392effd700 1 --2- 172.21.15.37:0/3417899757 >> v2:172.21.15.143:3300/0 conn(0x7f393013a480 0x7f39300a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:18:58.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392ffff700 1 --2- 172.21.15.37:0/3417899757 >> v2:172.21.15.37:3300/0 conn(0x7f39300a2040 0x7f39300a24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T07:18:58.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392f7fe700 1 --2- 172.21.15.37:0/3417899757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 0x7f3930139f40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f39300a33b0 tx=0x7f3920009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6962d6118a1e89e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 1 -- 172.21.15.37:0/3417899757 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392000b8a0 con 0x7f3930139b60 2024-09-16T07:18:58.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:58.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 10 monclient: shutdown 2024-09-16T07:18:58.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 1 -- 172.21.15.37:0/3044728284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f12141490d0 msgr2=0x7f121414b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f121c15c700 1 --2- 172.21.15.37:0/3044728284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f12141490d0 0x7f121414b4c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1214067530 tx=0x7f1210009bf0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 10 monclient: dump: 2024-09-16T07:18:58.741 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.741 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.741 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.741 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 5 monclient: authenticate success, global_id 4253 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 1 -- 172.21.15.37:0/3417899757 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392000ba00 con 0x7f3930139b60 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 1 -- 172.21.15.37:0/3417899757 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392000bd90 con 0x7f3930139b60 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f3934c46700 4 set_mon_vals no callback set 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.660+0000 7f392e7fc700 10 monclient: dump: 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.743 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.744 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.744 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 -- 172.21.15.37:0/3044728284 shutdown_connections 2024-09-16T07:18:58.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 --2- 172.21.15.37:0/3044728284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f12141490d0 0x7f121414b4c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 -- 172.21.15.37:0/3044728284 >> 172.21.15.37:0/3044728284 conn(0x7f12140a04b0 msgr2=0x7f12140a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f3934c46700 10 monclient: shutdown 2024-09-16T07:18:58.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f3934c46700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f3934c46700 1 -- 172.21.15.37:0/3417899757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 msgr2=0x7f3930139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f3934c46700 1 --2- 172.21.15.37:0/3417899757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 0x7f3930139f40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f39300a33b0 tx=0x7f3920009bf0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 -- 172.21.15.37:0/3044728284 shutdown_connections 2024-09-16T07:18:58.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f3934c46700 1 -- 172.21.15.37:0/3417899757 shutdown_connections 2024-09-16T07:18:58.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f3934c46700 1 --2- 172.21.15.37:0/3417899757 >> v2:172.21.15.37:3300/0 conn(0x7f39300a2040 0x7f39300a24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f3934c46700 1 --2- 172.21.15.37:0/3417899757 >> v2:172.21.15.143:3300/0 conn(0x7f393013a480 0x7f39300a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f3934c46700 1 --2- 172.21.15.37:0/3417899757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 0x7f3930139f40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f3934c46700 1 -- 172.21.15.37:0/3417899757 >> 172.21.15.37:0/3417899757 conn(0x7f39300a3c60 msgr2=0x7f39300a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 -- 172.21.15.37:0/3044728284 wait complete. 2024-09-16T07:18:58.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient: monmap: 2024-09-16T07:18:58.748 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 Processor -- start 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 -- start start 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient: init 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f121409cd40 0x7f12140893c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient(hunting): picked mon.noname-a con 0x7f121409cd40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f12141490d0 0x7f1214089900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient(hunting): picked mon.noname-c con 0x7f12141490d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f121414ba00 0x7f121413dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient(hunting): picked mon.noname-b con 0x7f121414ba00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12140a2fc0 con 0x7f121409cd40 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121414eea0 con 0x7f121414ba00 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1214072660 con 0x7f12141490d0 2024-09-16T07:18:58.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121c15c700 10 monclient(hunting): authenticate will time out at 859.154663s 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f121409cd40 0x7f12140893c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f121409cd40 0x7f12140893c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36448/0 (socket says 172.21.15.37:36448) 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 1 -- 172.21.15.37:0/3686138693 learned_addr learned my addr 172.21.15.37:0/3686138693 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 10 monclient(hunting): get_auth_request con 0x7f121409cd40 auth_method 0 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f12196f7700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f12141490d0 0x7f1214089900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f12196f7700 10 monclient(hunting): get_auth_request con 0x7f12141490d0 auth_method 0 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f12196f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f12196f7700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f12196f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121a6f9700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f121414ba00 0x7f121413dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121a6f9700 10 monclient(hunting): get_auth_request con 0x7f121414ba00 auth_method 0 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121a6f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121a6f9700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121a6f9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121a6f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121a6f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f121a6f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-09-16T07:18:58.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 1 -- 172.21.15.37:0/3686138693 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f12141490d0 msgr2=0x7f1214089900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f12141490d0 0x7f1214089900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 1 -- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f121414ba00 msgr2=0x7f121413dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f121414ba00 0x7f121413dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 1 -- 172.21.15.37:0/3686138693 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1210009570 con 0x7f121409cd40 2024-09-16T07:18:58.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.664+0000 7f1219ef8700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f121409cd40 0x7f12140893c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f120c001f70 tx=0x7f120c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 -- 172.21.15.37:0/3417899757 shutdown_connections 2024-09-16T07:18:58.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 -- 172.21.15.37:0/3417899757 wait complete. 2024-09-16T07:18:58.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient: monmap: 2024-09-16T07:18:58.757 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.757 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:58.757 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:58.757 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 Processor -- start 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 -- start start 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient: init 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39300a2040 0x7f39300702d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient(hunting): picked mon.noname-b con 0x7f39300a2040 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 0x7f39301379b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient(hunting): picked mon.noname-c con 0x7f3930139b60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f393013a480 0x7f3930137ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient(hunting): picked mon.noname-a con 0x7f393013a480 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3930124e90 con 0x7f393013a480 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39301387e0 con 0x7f39300a2040 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3930125370 con 0x7f3930139b60 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient(hunting): authenticate will time out at 859.157951s 2024-09-16T07:18:58.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39300a2040 0x7f39300702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392effd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 0x7f39301379b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39300a2040 0x7f39300702d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.37:51372/0 (socket says 172.21.15.37:51372) 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 1 -- 172.21.15.37:0/1828571177 learned_addr learned my addr 172.21.15.37:0/1828571177 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 10 monclient(hunting): get_auth_request con 0x7f39300a2040 auth_method 0 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392effd700 10 monclient(hunting): get_auth_request con 0x7f3930139b60 auth_method 0 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392ffff700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f393013a480 0x7f3930137ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392ffff700 10 monclient(hunting): get_auth_request con 0x7f393013a480 auth_method 0 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 10 monclient(hunting): handle_auth_done global_id 4291 payload 824 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 1 -- 172.21.15.37:0/1828571177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 msgr2=0x7f39301379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3930139b60 0x7f39301379b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 1 -- 172.21.15.37:0/1828571177 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f393013a480 msgr2=0x7f3930137ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f393013a480 0x7f3930137ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 1 monclient: found mon.noname-b 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 1 -- 172.21.15.37:0/1828571177 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920009570 con 0x7f39300a2040 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392ffff700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f393013a480 0x7f3930137ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392f7fe700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39300a2040 0x7f39300702d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f393009df30 tx=0x7f3920004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 1 -- 172.21.15.37:0/1828571177 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200045a0 con 0x7f39300a2040 2024-09-16T07:18:58.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 10 monclient: dump: 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 1 -- 172.21.15.37:0/1828571177 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392001f070 con 0x7f39300a2040 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 1 -- 172.21.15.37:0/1828571177 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920003900 con 0x7f39300a2040 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f392d7fa700 4 set_mon_vals no callback set 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3917fff700 10 monclient: dump: 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.764 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 5 monclient: authenticate success, global_id 4291 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient: _renew_subs 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39300724c0 con 0x7f39300a2040 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient: _renew_subs 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:18:58.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.668+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3930125090 con 0x7f39300a2040 2024-09-16T07:18:58.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.672+0000 7f3917fff700 1 -- 172.21.15.37:0/1828571177 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f3920021030 con 0x7f39300a2040 2024-09-16T07:18:58.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.672+0000 7f3917fff700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f39180357b0 0x7f3918037c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.672+0000 7f3917fff700 1 -- 172.21.15.37:0/1828571177 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7f3920048f50 con 0x7f39300a2040 2024-09-16T07:18:58.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.672+0000 7f392effd700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f39180357b0 0x7f3918037c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.672+0000 7f392effd700 10 monclient: get_auth_request con 0x7f39180357b0 auth_method 0 2024-09-16T07:18:58.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.672+0000 7f392effd700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f39180357b0 0x7f3918037c70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3930125580 tx=0x7f3928009910 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.767 INFO:teuthology.orchestra.run.smithi037.stdout:51539607556 2024-09-16T07:18:58.767 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T07:18:58.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 1 -- 172.21.15.37:0/3686138693 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c00ad80 con 0x7f121409cd40 2024-09-16T07:18:58.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:58.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 10 monclient: dump: 2024-09-16T07:18:58.769 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.769 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 1 -- 172.21.15.37:0/3686138693 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c0020c0 con 0x7f121409cd40 2024-09-16T07:18:58.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121c15c700 5 monclient: authenticate success, global_id 4278 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121c15c700 10 monclient: _renew_subs 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121c15c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1214141b80 con 0x7f121409cd40 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 1 -- 172.21.15.37:0/3686138693 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c012660 con 0x7f121409cd40 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 10 monclient: dump: 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.771 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121c15c700 10 monclient: _renew_subs 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121c15c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121413fa90 con 0x7f121409cd40 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120b7fe700 4 set_mon_vals no callback set 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 1 -- 172.21.15.37:0/3686138693 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f120c003a20 con 0x7f121409cd40 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121c15c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f11fc000f10 con 0x7f121409cd40 2024-09-16T07:18:58.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f11f8035650 0x7f11f8037b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 1 -- 172.21.15.37:0/3686138693 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7f120c047c20 con 0x7f121409cd40 2024-09-16T07:18:58.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] conn(0x7f11f803b460 0x7f11f803d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 1 -- 172.21.15.37:0/3686138693 --> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11f803df70 con 0x7f11f803b460 2024-09-16T07:18:58.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 1 -- 172.21.15.37:0/3686138693 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f120c005080 con 0x7f121409cd40 2024-09-16T07:18:58.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f120a7fc700 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-16T07:18:58.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121a6f9700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] conn(0x7f11f803b460 0x7f11f803d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121a6f9700 10 monclient: get_auth_request con 0x7f11f803b460 auth_method 0 2024-09-16T07:18:58.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7f121a6f9700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] conn(0x7f11f803b460 0x7f11f803d8a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 --> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc71011f780 con 0x7fc7101359e0 2024-09-16T07:18:58.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.676+0000 7fc70d7fa700 1 -- 172.21.15.37:0/2147210125 <== osd.0 v2:172.21.15.37:6800/3137250735 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc71011f780 con 0x7fc7101359e0 2024-09-16T07:18:58.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7f120a7fc700 1 -- 172.21.15.37:0/3686138693 <== osd.3 v2:172.21.15.143:6809/2588720019 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f11f803df70 con 0x7f11f803b460 2024-09-16T07:18:58.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 -- 172.21.15.37:0/1269860767 --> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff064005c20 con 0x7ff0640015e0 2024-09-16T07:18:58.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7f3934c46700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] conn(0x7f39100015e0 0x7f3910003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 --> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3910006b40 con 0x7f39100015e0 2024-09-16T07:18:58.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 --> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faebc005bf0 con 0x7faebc0015e0 2024-09-16T07:18:58.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x7fc7101359e0 msgr2=0x7fc71011c420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x7fc7101359e0 0x7fc71011c420 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc6f4035600 msgr2=0x7fc6f4037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc6f4035600 0x7fc6f4037ac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc7040090b0 tx=0x7fc704008030 comp rx=0 tx=0).stop 2024-09-16T07:18:58.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 10 monclient: shutdown 2024-09-16T07:18:58.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc710071170 msgr2=0x7fc7101ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 1 --2- 172.21.15.37:0/2147210125 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc710071170 0x7fc7101ca4a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc70800caf0 tx=0x7fc70800cb20 comp rx=0 tx=0).stop 2024-09-16T07:18:58.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7f392ffff700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] conn(0x7f39100015e0 0x7f3910003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc715297700 1 -- 172.21.15.37:0/2147210125 reap_dead start 2024-09-16T07:18:58.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faeceffd700 1 -- 172.21.15.37:0/1005645073 <== osd.2 v2:172.21.15.37:6816/265940208 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7faebc005bf0 con 0x7faebc0015e0 2024-09-16T07:18:58.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 shutdown_connections 2024-09-16T07:18:58.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 >> 172.21.15.37:0/2147210125 conn(0x7fc7100a04b0 msgr2=0x7fc71009ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7f392ffff700 10 monclient: get_auth_request con 0x7f39100015e0 auth_method 0 2024-09-16T07:18:58.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 shutdown_connections 2024-09-16T07:18:58.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff07a7fc700 1 -- 172.21.15.37:0/1269860767 <== osd.4 v2:172.21.15.143:6808/3355421705 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff064005c20 con 0x7ff0640015e0 2024-09-16T07:18:58.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7fc716cfa700 1 -- 172.21.15.37:0/2147210125 wait complete. 2024-09-16T07:18:58.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] conn(0x7faebc0015e0 msgr2=0x7faebc003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] conn(0x7faebc0015e0 0x7faebc003aa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 -- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] conn(0x7ff0640015e0 msgr2=0x7ff064003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] conn(0x7ff0640015e0 0x7ff064003aa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faec4035760 msgr2=0x7faec4037c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faec4035760 0x7faec4037c20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7faed000d6f0 tx=0x7faed0012000 comp rx=0 tx=0).stop 2024-09-16T07:18:58.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 10 monclient: shutdown 2024-09-16T07:18:58.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faed81369c0 msgr2=0x7faed8122f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 --2- 172.21.15.37:0/1005645073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faed81369c0 0x7faed8122f10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faec800c7f0 tx=0x7faec800cb00 comp rx=0 tx=0).stop 2024-09-16T07:18:58.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faede81a700 1 -- 172.21.15.37:0/1005645073 reap_dead start 2024-09-16T07:18:58.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 -- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff070035760 msgr2=0x7ff070037c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff070035760 0x7ff070037c20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff084132100 tx=0x7ff074007ee0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 10 monclient: shutdown 2024-09-16T07:18:58.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 -- 172.21.15.37:0/1269860767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff08409d9d0 msgr2=0x7ff0841309f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 --2- 172.21.15.37:0/1269860767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff08409d9d0 0x7ff0841309f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff080009150 tx=0x7ff080004770 comp rx=0 tx=0).stop 2024-09-16T07:18:58.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff08a005700 1 -- 172.21.15.37:0/1269860767 reap_dead start 2024-09-16T07:18:58.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 -- 172.21.15.37:0/1269860767 shutdown_connections 2024-09-16T07:18:58.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 -- 172.21.15.37:0/1269860767 >> 172.21.15.37:0/1269860767 conn(0x7ff0840a3c60 msgr2=0x7ff0840a2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 shutdown_connections 2024-09-16T07:18:58.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 >> 172.21.15.37:0/1005645073 conn(0x7faed80a04b0 msgr2=0x7faed806ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7f392ffff700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] conn(0x7f39100015e0 0x7f3910003aa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 -- 172.21.15.37:0/1269860767 shutdown_connections 2024-09-16T07:18:58.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 shutdown_connections 2024-09-16T07:18:58.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7ff06ffff700 1 -- 172.21.15.37:0/1269860767 wait complete. 2024-09-16T07:18:58.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.680+0000 7faee027d700 1 -- 172.21.15.37:0/1005645073 wait complete. 2024-09-16T07:18:58.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.684+0000 7f3917fff700 1 -- 172.21.15.37:0/1828571177 <== osd.5 v2:172.21.15.143:6800/3921632375 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f3910006b40 con 0x7f39100015e0 2024-09-16T07:18:58.784 INFO:teuthology.orchestra.run.smithi037.stdout:55834574854 2024-09-16T07:18:58.785 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T07:18:58.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 --> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f11fc002c30 con 0x7f11f803b460 2024-09-16T07:18:58.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f120a7fc700 1 -- 172.21.15.37:0/3686138693 <== osd.3 v2:172.21.15.143:6809/2588720019 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f11fc002c30 con 0x7f11f803b460 2024-09-16T07:18:58.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] conn(0x7f11f803b460 msgr2=0x7f11f803d8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f121c15c700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] conn(0x7f11f803b460 0x7f11f803d8a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f11f8035650 msgr2=0x7f11f8037b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f121c15c700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f11f8035650 0x7f11f8037b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f121c15c700 10 monclient: shutdown 2024-09-16T07:18:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f121c15c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f121409cd40 msgr2=0x7f12140893c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f121c15c700 1 --2- 172.21.15.37:0/3686138693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f121409cd40 0x7f12140893c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f120c001f70 tx=0x7f120c00a340 comp rx=0 tx=0).stop 2024-09-16T07:18:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.696+0000 7f121a6f9700 1 -- 172.21.15.37:0/3686138693 reap_dead start 2024-09-16T07:18:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 shutdown_connections 2024-09-16T07:18:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 >> 172.21.15.37:0/3686138693 conn(0x7f12140a04b0 msgr2=0x7f121409bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 --> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3910005c20 con 0x7f39100015e0 2024-09-16T07:18:58.788 INFO:teuthology.orchestra.run.smithi037.stdout:55834574854 2024-09-16T07:18:58.788 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T07:18:58.789 INFO:teuthology.orchestra.run.smithi037.stdout:51539607557 2024-09-16T07:18:58.789 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T07:18:58.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 shutdown_connections 2024-09-16T07:18:58.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f121c15c700 1 -- 172.21.15.37:0/3686138693 wait complete. 2024-09-16T07:18:58.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3917fff700 1 -- 172.21.15.37:0/1828571177 <== osd.5 v2:172.21.15.143:6800/3921632375 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3910005c20 con 0x7f39100015e0 2024-09-16T07:18:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] conn(0x7f39100015e0 msgr2=0x7f3910003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] conn(0x7f39100015e0 0x7f3910003aa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f39180357b0 msgr2=0x7f3918037c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f39180357b0 0x7f3918037c70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3930125580 tx=0x7f3928009910 comp rx=0 tx=0).stop 2024-09-16T07:18:58.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 10 monclient: shutdown 2024-09-16T07:18:58.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39300a2040 msgr2=0x7f39300702d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 --2- 172.21.15.37:0/1828571177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39300a2040 0x7f39300702d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f393009df30 tx=0x7f3920004770 comp rx=0 tx=0).stop 2024-09-16T07:18:58.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f392ffff700 1 -- 172.21.15.37:0/1828571177 reap_dead start 2024-09-16T07:18:58.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 shutdown_connections 2024-09-16T07:18:58.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 >> 172.21.15.37:0/1828571177 conn(0x7f39300a3c60 msgr2=0x7f3930138d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 shutdown_connections 2024-09-16T07:18:58.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.700+0000 7f3934c46700 1 -- 172.21.15.37:0/1828571177 wait complete. 2024-09-16T07:18:58.793 INFO:teuthology.orchestra.run.smithi037.stdout:55834574853 2024-09-16T07:18:58.793 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T07:18:58.794 INFO:teuthology.orchestra.run.smithi037.stdout:47244640260 2024-09-16T07:18:58.794 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T07:18:58.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient: monmap: 2024-09-16T07:18:58.973 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.976969+0000 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.976969+0000 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 1 Processor -- start 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 1 -- start start 2024-09-16T07:18:58.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient: init 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdf400a2360 0x7fdf400a2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient(hunting): picked mon.noname-c con 0x7fdf400a2360 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf400a2c80 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fdf40071590 0x7fdf40135830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient(hunting): picked mon.noname-a con 0x7fdf40071590 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf400a33c0 con 0x7fdf400a2c80 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf40099d70 con 0x7fdf40071590 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4009a050 con 0x7fdf400a2360 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf44a67700 10 monclient(hunting): authenticate will time out at 859.463115s 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fdf40071590 0x7fdf40135830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fdf40071590 0x7fdf40135830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36472/0 (socket says 172.21.15.37:36472) 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 1 -- 172.21.15.37:0/921909814 learned_addr learned my addr 172.21.15.37:0/921909814 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 10 monclient(hunting): get_auth_request con 0x7fdf40071590 auth_method 0 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3f7fe700 1 --2- 172.21.15.37:0/921909814 >> v2:172.21.15.143:3300/0 conn(0x7fdf400a2360 0x7fdf400a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3f7fe700 10 monclient(hunting): get_auth_request con 0x7fdf400a2360 auth_method 0 2024-09-16T07:18:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7f5d208af700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7f5d208af700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7f5d208af700 10 monclient: monmap: 2024-09-16T07:18:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.979 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.978143+0000 2024-09-16T07:18:58.979 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.978143+0000 2024-09-16T07:18:58.979 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.979 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.979 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.979 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.979 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.979 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.979 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.979 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-09-16T07:18:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 1 -- 172.21.15.37:0/921909814 >> v2:172.21.15.143:3300/0 conn(0x7fdf400a2360 msgr2=0x7fdf400a2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 1 --2- 172.21.15.37:0/921909814 >> v2:172.21.15.143:3300/0 conn(0x7fdf400a2360 0x7fdf400a2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 1 -- 172.21.15.37:0/921909814 >> v1:172.21.15.143:6789/0 conn(0x7fdf400a2c80 legacy=0x7fdf40135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fdf3ffff700 1 -- 172.21.15.37:0/921909814 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf400a1fe0 con 0x7fdf40071590 2024-09-16T07:18:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fd062274700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fd062274700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7fd062274700 10 monclient: monmap: 2024-09-16T07:18:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.978343+0000 2024-09-16T07:18:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.978343+0000 2024-09-16T07:18:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.982 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.982 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.982 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.982 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.982 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.972+0000 7f5d208af700 1 Processor -- start 2024-09-16T07:18:58.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3ffff700 1 --2- 172.21.15.37:0/921909814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 0x7fdf40135830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdf400a48b0 tx=0x7fdf3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df690c7dd5c58c13 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 -- start start 2024-09-16T07:18:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient: init 2024-09-16T07:18:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 1 Processor -- start 2024-09-16T07:18:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 1 -- start start 2024-09-16T07:18:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 10 monclient: init 2024-09-16T07:18:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 1 -- 172.21.15.37:0/921909814 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3800e070 con 0x7fdf40071590 2024-09-16T07:18:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 10 monclient: dump: 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 10 monclient: _finish_auth 0 2024-09-16T07:18:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 5 monclient: authenticate success, global_id 4284 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 1 -- 172.21.15.37:0/921909814 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf38003a40 con 0x7fdf40071590 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 4 set_mon_vals no callback set 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 1 -- 172.21.15.37:0/921909814 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf38003df0 con 0x7fdf40071590 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3effd700 10 monclient: dump: 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:58.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.987 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.987 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:58.987 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:58.987 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.987 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:58.987 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:58.987 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:58.987 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5d1c0707a0 0x7f5d1c0a2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient(hunting): picked mon.noname-a con 0x7f5d1c0707a0 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d1c071680 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d1c0a2a50 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1c070f80 con 0x7f5d1c0a2a50 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1c0a2100 con 0x7f5d1c071680 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1c09a3c0 con 0x7f5d1c0707a0 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient(hunting): authenticate will time out at 859.464858s 2024-09-16T07:18:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5d1c0707a0 0x7f5d1c0a2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5d1c0707a0 0x7f5d1c0a2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36476/0 (socket says 172.21.15.37:36476) 2024-09-16T07:18:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 1 -- 172.21.15.37:0/1664142287 learned_addr learned my addr 172.21.15.37:0/1664142287 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1b59e700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f5d1c0a2a50 0x7f5d1c135830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36626/0 (socket says 172.21.15.37:36626) 2024-09-16T07:18:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 10 monclient(hunting): get_auth_request con 0x7f5d1c0707a0 auth_method 0 2024-09-16T07:18:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f568b59e700 10 monclient: get_monmap_and_config 2024-09-16T07:18:58.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient: shutdown 2024-09-16T07:18:58.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:58.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 -- 172.21.15.37:0/921909814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 msgr2=0x7fdf40135830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 --2- 172.21.15.37:0/921909814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 0x7fdf40135830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdf400a48b0 tx=0x7fdf3800b330 comp rx=0 tx=0).stop 2024-09-16T07:18:58.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f568b59e700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 -- 172.21.15.37:0/921909814 shutdown_connections 2024-09-16T07:18:58.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 --2- 172.21.15.37:0/921909814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 0x7fdf40135830 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 --2- 172.21.15.37:0/921909814 >> v2:172.21.15.143:3300/0 conn(0x7fdf400a2360 0x7fdf400a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:58.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 -- 172.21.15.37:0/921909814 >> 172.21.15.37:0/921909814 conn(0x7fdf400a3c60 msgr2=0x7fdf400a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:58.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f568b59e700 10 monclient: monmap: 2024-09-16T07:18:58.992 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.992 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:58.992 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.980700+0000 2024-09-16T07:18:58.992 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.980700+0000 2024-09-16T07:18:58.992 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.992 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:58.992 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:58.993 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:58.993 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:58.993 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:58.993 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:58.993 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:58.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:58.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 -- 172.21.15.37:0/921909814 shutdown_connections 2024-09-16T07:18:58.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:58.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 -- 172.21.15.37:0/921909814 wait complete. 2024-09-16T07:18:58.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 1 -- 172.21.15.37:0/1664142287 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066424079 0 0) 0x7f5d1c070f80 con 0x7f5d1c0a2a50 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient(hunting): _negotiate 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient(hunting): my global_id is 4287 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 1 -- 172.21.15.37:0/1664142287 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d040035d0 con 0x7f5d1c0a2a50 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 1 -- 172.21.15.37:0/1664142287 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027953695 0 0) 0x7f5d1c0a2100 con 0x7f5d1c071680 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient(hunting): _negotiate 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient(hunting): my global_id is 4297 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 1 -- 172.21.15.37:0/1664142287 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d1c070f80 con 0x7f5d1c071680 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 10 monclient(hunting): handle_auth_done global_id 4290 payload 274 2024-09-16T07:18:58.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 1 -- 172.21.15.37:0/1664142287 >> v1:172.21.15.143:6789/0 conn(0x7f5d1c071680 legacy=0x7f5d1c135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 1 -- 172.21.15.37:0/1664142287 >> v1:172.21.15.37:6789/0 conn(0x7f5d1c0a2a50 legacy=0x7f5d1c135830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:58.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 10 monclient: _finish_hunting 0 2024-09-16T07:18:58.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 1 monclient: found mon.noname-a 2024-09-16T07:18:58.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:58.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:58.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 1 -- 172.21.15.37:0/1664142287 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1c135f40 con 0x7f5d1c0707a0 2024-09-16T07:18:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T07:18:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd05c071170 0x7fd05c071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 10 monclient(hunting): picked mon.noname-a con 0x7fd05c071170 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd05c09fa00 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd05c071a90 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c09d6b0 con 0x7fd05c071a90 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c1361b0 con 0x7fd05c09fa00 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05c099de0 con 0x7fd05c071170 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd062274700 10 monclient(hunting): authenticate will time out at 859.466238s 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd05c071170 0x7fd05c071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient: build_initial_monmap 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient: monmap: 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:58.998 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:58.999 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:58.999 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:58.999 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:58.999 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:58.999 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:58.999 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:58.999 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:58.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d1ad9d700 1 --2- 172.21.15.37:0/1664142287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c0707a0 0x7f5d1c0a2510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009910 tx=0x7f5d10009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f88b2d8110aeb45 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd05c071170 0x7fd05c071550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36486/0 (socket says 172.21.15.37:36486) 2024-09-16T07:18:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 1 -- 172.21.15.37:0/1846218951 learned_addr learned my addr 172.21.15.37:0/1846218951 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 10 monclient(hunting): get_auth_request con 0x7fd05c071170 auth_method 0 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 1 -- 172.21.15.37:0/1846218951 >> v1:172.21.15.143:6789/0 conn(0x7fd05c09fa00 legacy=0x7fd05c135390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 1 -- 172.21.15.37:0/1846218951 >> v1:172.21.15.37:6789/0 conn(0x7fd05c071a90 legacy=0x7fd05c135aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 10 monclient: _finish_hunting 0 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 1 monclient: found mon.noname-a 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 1 -- 172.21.15.37:0/1846218951 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c09a0c0 con 0x7fd05c071170 2024-09-16T07:18:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 1 -- 172.21.15.37:0/1664142287 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d10011830 con 0x7f5d1c0707a0 2024-09-16T07:18:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient: dump: 2024-09-16T07:18:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 1 -- 172.21.15.37:0/1664142287 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d10011990 con 0x7f5d1c0707a0 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 1 -- 172.21.15.37:0/1664142287 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d10011d40 con 0x7f5d1c0707a0 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 5 monclient: authenticate success, global_id 4290 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d19d9b700 10 monclient: dump: 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.005 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 4 set_mon_vals no callback set 2024-09-16T07:18:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fd05bfff700 1 --2- 172.21.15.37:0/1846218951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c071170 0x7fd05c071550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd054009e50 tx=0x7fd05400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468048a38641f8ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 Processor -- start 2024-09-16T07:18:59.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 -- start start 2024-09-16T07:18:59.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient: init 2024-09-16T07:18:59.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:59.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 0x7fdf401ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient(hunting): picked mon.noname-a con 0x7fdf40071590 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf400a2360 0x7fdf401ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient(hunting): picked mon.noname-c con 0x7fdf400a2360 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf400a2c80 0x7fdf40121190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient(hunting): picked mon.noname-b con 0x7fdf400a2c80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf40123b10 con 0x7fdf40071590 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4009ec50 con 0x7fdf400a2c80 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf400724c0 con 0x7fdf400a2360 2024-09-16T07:18:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf44a67700 10 monclient(hunting): authenticate will time out at 859.467682s 2024-09-16T07:18:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 10 monclient: shutdown 2024-09-16T07:18:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 0x7fdf401ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 -- 172.21.15.37:0/1664142287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c0707a0 msgr2=0x7f5d1c0a2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 --2- 172.21.15.37:0/1664142287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c0707a0 0x7f5d1c0a2510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009910 tx=0x7f5d10009c20 comp rx=0 tx=0).stop 2024-09-16T07:18:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 0x7fdf401ca230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36498/0 (socket says 172.21.15.37:36498) 2024-09-16T07:18:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3f7fe700 1 -- 172.21.15.37:0/1140362169 learned_addr learned my addr 172.21.15.37:0/1140362169 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3f7fe700 10 monclient(hunting): get_auth_request con 0x7fdf40071590 auth_method 0 2024-09-16T07:18:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 -- 172.21.15.37:0/1664142287 shutdown_connections 2024-09-16T07:18:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 --2- 172.21.15.37:0/1664142287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c0707a0 0x7f5d1c0a2510 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 -- 172.21.15.37:0/1664142287 >> 172.21.15.37:0/1664142287 conn(0x7f5d1c0a3c60 msgr2=0x7f5d1c0a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf37fff700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf400a2360 0x7fdf401ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf37fff700 10 monclient(hunting): get_auth_request con 0x7fdf400a2360 auth_method 0 2024-09-16T07:18:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf37fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf37fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf37fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7f5d208af700 1 -- 172.21.15.37:0/1664142287 shutdown_connections 2024-09-16T07:18:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.976+0000 7fdf3f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3ffff700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf400a2c80 0x7fdf40121190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf37fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 -- 172.21.15.37:0/1664142287 wait complete. 2024-09-16T07:18:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf37fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient: build_initial_monmap 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient: monmap: 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:59.013 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 1 Processor -- start 2024-09-16T07:18:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 1 -- start start 2024-09-16T07:18:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 10 monclient: init 2024-09-16T07:18:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 Processor -- start 2024-09-16T07:18:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 -- start start 2024-09-16T07:18:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient: init 2024-09-16T07:18:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:18:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1c0707a0 0x7f5d1c1ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient(hunting): picked mon.noname-b con 0x7f5d1c0707a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c071680 0x7f5d1c1ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient(hunting): picked mon.noname-a con 0x7f5d1c071680 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2a50 0x7f5d1c120b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T07:18:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient(hunting): picked mon.noname-c con 0x7f5d1c0a2a50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c0a08f0 con 0x7f5d1c071680 2024-09-16T07:18:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c124770 con 0x7f5d1c0707a0 2024-09-16T07:18:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f568c0a0e90 addr v1:172.21.15.143:6789/0 2024-09-16T07:18:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f568c0a04d0 0x7f568c0a1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c1339f0 con 0x7f5d1c0a2a50 2024-09-16T07:18:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient(hunting): authenticate will time out at 859.469663s 2024-09-16T07:18:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f568c0a04d0 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a2d50 0x7f568c0a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f568c0a2d50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf37fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3ffff700 10 monclient(hunting): get_auth_request con 0x7fdf400a2c80 auth_method 0 2024-09-16T07:18:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2024-09-16T07:18:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 1 -- 172.21.15.37:0/1140362169 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf400a2360 msgr2=0x7fdf401ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf400a2360 0x7fdf401ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 1 -- 172.21.15.37:0/1140362169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf400a2c80 msgr2=0x7fdf40121190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568c0a0900 con 0x7f568c0a0e90 2024-09-16T07:18:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568c0a17b0 con 0x7f568c0a04d0 2024-09-16T07:18:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c06cf50 con 0x7f568c0a2d50 2024-09-16T07:18:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568b59e700 10 monclient(hunting): authenticate will time out at 859.469776s 2024-09-16T07:18:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c071680 0x7f5d1c1ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c071680 0x7f5d1c1ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36508/0 (socket says 172.21.15.37:36508) 2024-09-16T07:18:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 1 -- 172.21.15.37:0/362898632 learned_addr learned my addr 172.21.15.37:0/362898632 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 10 monclient(hunting): get_auth_request con 0x7f5d1c071680 auth_method 0 2024-09-16T07:18:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf400a2c80 0x7fdf40121190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:18:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 1 monclient: found mon.noname-a 2024-09-16T07:18:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 1 -- 172.21.15.37:0/1140362169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf38009b30 con 0x7fdf40071590 2024-09-16T07:18:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1ad9d700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1c0707a0 0x7f5d1c1ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1ad9d700 10 monclient(hunting): get_auth_request con 0x7f5d1c0707a0 auth_method 0 2024-09-16T07:18:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fecff59e700 10 monclient: get_monmap_and_config 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fecff59e700 10 monclient: build_initial_monmap 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fecff59e700 10 monclient: monmap: 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.984338+0000 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.984338+0000 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:59.023 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a2d50 0x7f568c0a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3f7fe700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 0x7fdf401ca230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800e790 tx=0x7fdf2800eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 1 -- 172.21.15.37:0/1846218951 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05400e070 con 0x7fd05c071170 2024-09-16T07:18:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-09-16T07:18:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 1 -- 172.21.15.37:0/362898632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2a50 msgr2=0x7f5d1c120b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 10 monclient: dump: 2024-09-16T07:18:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.026 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2a50 0x7f5d1c120b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 1 -- 172.21.15.37:0/362898632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1c0707a0 msgr2=0x7f5d1c1ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1c0707a0 0x7f5d1c1ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 10 monclient: _finish_hunting 0 2024-09-16T07:18:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 1 monclient: found mon.noname-a 2024-09-16T07:18:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 1 -- 172.21.15.37:0/1846218951 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05400b9c0 con 0x7fd05c071170 2024-09-16T07:18:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 1 -- 172.21.15.37:0/1846218951 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05400bd70 con 0x7fd05c071170 2024-09-16T07:18:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 1 -- 172.21.15.37:0/362898632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10009570 con 0x7f5d1c071680 2024-09-16T07:18:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd05affd700 10 monclient: dump: 2024-09-16T07:18:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.029 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd062274700 5 monclient: authenticate success, global_id 4293 2024-09-16T07:18:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd062274700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd062274700 4 set_mon_vals no callback set 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568a59c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f568c0a0e90 0x7f568c0a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.37:42366/0 (socket says 172.21.15.37:42366) 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a2d50 0x7f568c0a3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40006/0 (socket says 172.21.15.37:40006) 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568a59c700 1 -- 172.21.15.37:0/1698854083 learned_addr learned my addr 172.21.15.37:0/1698854083 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 10 monclient(hunting): get_auth_request con 0x7f568c0a2d50 auth_method 0 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5689d9b700 1 --2- 172.21.15.37:0/1698854083 >> v2:172.21.15.143:3300/0 conn(0x7f568c0a04d0 0x7f568c0a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 1 -- 172.21.15.37:0/1140362169 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28013610 con 0x7fdf40071590 2024-09-16T07:18:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 10 monclient: dump: 2024-09-16T07:18:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.032 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5689d9b700 10 monclient(hunting): get_auth_request con 0x7f568c0a04d0 auth_method 0 2024-09-16T07:18:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5689d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5689d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5689d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-09-16T07:18:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 1 -- 172.21.15.37:0/1698854083 >> v2:172.21.15.143:3300/0 conn(0x7f568c0a04d0 msgr2=0x7f568c0a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 1 --2- 172.21.15.37:0/1698854083 >> v2:172.21.15.143:3300/0 conn(0x7f568c0a04d0 0x7f568c0a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1a59c700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c071680 0x7f5d1c1ca7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400cc80 tx=0x7f5d1400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 1 -- 172.21.15.37:0/1140362169 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf28013770 con 0x7fdf40071590 2024-09-16T07:18:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 1 -- 172.21.15.37:0/1698854083 >> v1:172.21.15.143:6789/0 conn(0x7f568c0a0e90 legacy=0x7f568c0a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 10 monclient: _finish_hunting 0 2024-09-16T07:18:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 1 monclient: found mon.noname-b 2024-09-16T07:18:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3dffb700 4 set_mon_vals no callback set 2024-09-16T07:18:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 1 -- 172.21.15.37:0/1698854083 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c0a36c0 con 0x7f568c0a2d50 2024-09-16T07:18:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568ad9d700 1 --2- 172.21.15.37:0/1698854083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a2d50 0x7f568c0a3180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f568400b050 tx=0x7f568400b360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12972126b5a6deff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 1 -- 172.21.15.37:0/362898632 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1400ada0 con 0x7f5d1c071680 2024-09-16T07:18:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 10 monclient: dump: 2024-09-16T07:18:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 5 monclient: authenticate success, global_id 4302 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 1 -- 172.21.15.37:0/362898632 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1400ce80 con 0x7f5d1c071680 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 1 -- 172.21.15.37:0/362898632 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d14003c60 con 0x7f5d1c071680 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient: _renew_subs 2024-09-16T07:18:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 -- 172.21.15.37:0/362898632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1c0724c0 con 0x7f5d1c071680 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d1959a700 4 set_mon_vals no callback set 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d037fe700 10 monclient: dump: 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.039 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient: _renew_subs 2024-09-16T07:18:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f5d208af700 1 -- 172.21.15.37:0/362898632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1c123250 con 0x7f5d1c071680 2024-09-16T07:18:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd062274700 10 monclient: shutdown 2024-09-16T07:18:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd062274700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd062274700 1 -- 172.21.15.37:0/1846218951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c071170 msgr2=0x7fd05c071550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd062274700 1 --2- 172.21.15.37:0/1846218951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c071170 0x7fd05c071550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd054009e50 tx=0x7fd05400b330 comp rx=0 tx=0).stop 2024-09-16T07:18:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf37fff700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf400a2360 0x7fdf401ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:18:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf44a67700 5 monclient: authenticate success, global_id 4299 2024-09-16T07:18:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 1 -- 172.21.15.37:0/1140362169 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28013ae0 con 0x7fdf40071590 2024-09-16T07:18:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd062274700 1 -- 172.21.15.37:0/1846218951 shutdown_connections 2024-09-16T07:18:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd062274700 1 --2- 172.21.15.37:0/1846218951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c071170 0x7fd05c071550 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fd062274700 1 -- 172.21.15.37:0/1846218951 >> 172.21.15.37:0/1846218951 conn(0x7fd05c0a04b0 msgr2=0x7fd05c0a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf3cff9700 10 monclient: dump: 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf44a67700 10 monclient: _renew_subs 2024-09-16T07:18:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf44a67700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf44a67700 1 -- 172.21.15.37:0/1140362169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf400a1120 con 0x7fdf40071590 2024-09-16T07:18:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf44a67700 10 monclient: _renew_subs 2024-09-16T07:18:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf44a67700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7fdf44a67700 1 -- 172.21.15.37:0/1140362169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf40125110 con 0x7fdf40071590 2024-09-16T07:18:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fdf3cff9700 1 -- 172.21.15.37:0/1140362169 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fdf28013c40 con 0x7fdf40071590 2024-09-16T07:18:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fdf44a67700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fdf44a67700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fdf44a67700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fdf44a67700 1 -- 172.21.15.37:0/1140362169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf400a2740 con 0x7fdf40071590 2024-09-16T07:18:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 1 -- 172.21.15.37:0/1846218951 shutdown_connections 2024-09-16T07:18:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fdf3cff9700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdf20035650 0x7fdf20037b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 1 -- 172.21.15.37:0/1846218951 wait complete. 2024-09-16T07:18:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fdf3cff9700 1 -- 172.21.15.37:0/1140362169 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7fdf28047640 con 0x7fdf40071590 2024-09-16T07:18:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.980+0000 7f568959a700 1 -- 172.21.15.37:0/1698854083 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568400e070 con 0x7f568c0a2d50 2024-09-16T07:18:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 10 monclient: dump: 2024-09-16T07:18:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 1 -- 172.21.15.37:0/1698854083 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568400bb10 con 0x7f568c0a2d50 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 1 -- 172.21.15.37:0/1698854083 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684012430 con 0x7f568c0a2d50 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568959a700 10 monclient: dump: 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 5 monclient: authenticate success, global_id 4265 2024-09-16T07:18:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 4 set_mon_vals no callback set 2024-09-16T07:18:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient: shutdown 2024-09-16T07:18:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 -- 172.21.15.37:0/1698854083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a2d50 msgr2=0x7f568c0a3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 --2- 172.21.15.37:0/1698854083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a2d50 0x7f568c0a3180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f568400b050 tx=0x7f568400b360 comp rx=0 tx=0).stop 2024-09-16T07:18:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5d208af700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5d208af700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5d208af700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5d037fe700 1 -- 172.21.15.37:0/362898632 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f5d140050d0 con 0x7f5d1c071680 2024-09-16T07:18:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5d208af700 1 -- 172.21.15.37:0/362898632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1c0a2510 con 0x7f5d1c071680 2024-09-16T07:18:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5d037fe700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5d08035600 0x7f5d08037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5d037fe700 1 -- 172.21.15.37:0/362898632 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7f5d14047d50 con 0x7f5d1c071680 2024-09-16T07:18:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5d1ad9d700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5d08035600 0x7f5d08037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 -- 172.21.15.37:0/1698854083 shutdown_connections 2024-09-16T07:18:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5d1ad9d700 10 monclient: get_auth_request con 0x7f5d08035600 auth_method 0 2024-09-16T07:18:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 --2- 172.21.15.37:0/1698854083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a2d50 0x7f568c0a3180 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 --2- 172.21.15.37:0/1698854083 >> v2:172.21.15.143:3300/0 conn(0x7f568c0a04d0 0x7f568c0a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 -- 172.21.15.37:0/1698854083 >> 172.21.15.37:0/1698854083 conn(0x7f568c0a4740 msgr2=0x7f568c0a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fdf37fff700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdf20035650 0x7fdf20037b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fdf37fff700 10 monclient: get_auth_request con 0x7fdf20035650 auth_method 0 2024-09-16T07:18:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 -- 172.21.15.37:0/1698854083 shutdown_connections 2024-09-16T07:18:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 -- 172.21.15.37:0/1698854083 wait complete. 2024-09-16T07:18:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient: build_initial_monmap 2024-09-16T07:18:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient: monmap: 2024-09-16T07:18:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:59.052 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5d1ad9d700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5d08035600 0x7f5d08037ac0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5d10006f80 tx=0x7f5d1001c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fdf37fff700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdf20035650 0x7fdf20037b10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf4009ca30 tx=0x7fdf30009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient: build_initial_monmap 2024-09-16T07:18:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient: monmap: 2024-09-16T07:18:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:59.054 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 1 Processor -- start 2024-09-16T07:18:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 1 -- start start 2024-09-16T07:18:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient: init 2024-09-16T07:18:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 Processor -- start 2024-09-16T07:18:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 -- start start 2024-09-16T07:18:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient: init 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c071170 0x7fd05c134c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient(hunting): picked mon.noname-a con 0x7fd05c071170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071a90 0x7fd05c135190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient(hunting): picked mon.noname-c con 0x7fd05c071a90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd05c09fa00 0x7fd05c11ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient(hunting): picked mon.noname-b con 0x7fd05c09fa00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c11e310 con 0x7fd05c071170 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c11e7f0 con 0x7fd05c09fa00 2024-09-16T07:18:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c11d2d0 con 0x7fd05c071a90 2024-09-16T07:18:59.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:59.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd062274700 10 monclient(hunting): authenticate will time out at 859.475109s 2024-09-16T07:18:59.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:59.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:18:59.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f568c0a04d0 0x7f568c1cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f568c0a04d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:59.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a0e90 0x7f568c1c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f568c0a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f568c0a2d50 0x7f568c1ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f568c0a2d50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c06ef50 con 0x7f568c0a04d0 2024-09-16T07:18:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c09df70 con 0x7f568c0a2d50 2024-09-16T07:18:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c06bfa0 con 0x7f568c0a0e90 2024-09-16T07:18:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:59.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568b59e700 10 monclient(hunting): authenticate will time out at 859.475234s 2024-09-16T07:18:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071a90 0x7fd05c135190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071a90 0x7fd05c135190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40014/0 (socket says 172.21.15.37:40014) 2024-09-16T07:18:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 1 -- 172.21.15.37:0/3401530999 learned_addr learned my addr 172.21.15.37:0/3401530999 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 10 monclient(hunting): get_auth_request con 0x7fd05c071a90 auth_method 0 2024-09-16T07:18:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05bfff700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c071170 0x7fd05c134c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05bfff700 10 monclient(hunting): get_auth_request con 0x7fd05c071170 auth_method 0 2024-09-16T07:18:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5689d9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a0e90 0x7f568c1c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5689d9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a0e90 0x7f568c1c8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40030/0 (socket says 172.21.15.37:40030) 2024-09-16T07:18:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5689d9b700 1 -- 172.21.15.37:0/1515810832 learned_addr learned my addr 172.21.15.37:0/1515810832 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5689d9b700 10 monclient(hunting): get_auth_request con 0x7f568c0a0e90 auth_method 0 2024-09-16T07:18:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5689d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5689d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5689d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd060811700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd05c09fa00 0x7fd05c11ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 1 -- 172.21.15.37:0/3401530999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd05c09fa00 msgr2=0x7fd05c11ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd05c09fa00 0x7fd05c11ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 1 -- 172.21.15.37:0/3401530999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c071170 msgr2=0x7fd05c134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c071170 0x7fd05c134c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 1 monclient: found mon.noname-c 2024-09-16T07:18:59.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:59.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7fd05b7fe700 1 -- 172.21.15.37:0/3401530999 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd054009b30 con 0x7fd05c071a90 2024-09-16T07:18:59.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd05b7fe700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071a90 0x7fd05c135190 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd05000c6b0 tx=0x7fd05000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 1 Processor -- start 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568ad9d700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f568c0a2d50 0x7f568c1ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5689d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5689d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f5689d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568ad9d700 10 monclient(hunting): get_auth_request con 0x7f568c0a2d50 auth_method 0 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568a59c700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f568c0a04d0 0x7f568c1cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568a59c700 10 monclient(hunting): get_auth_request con 0x7f568c0a04d0 auth_method 0 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.984+0000 7f568a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 1 -- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f568c0a2d50 msgr2=0x7f568c1ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f568c0a2d50 0x7f568c1ce2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 1 -- 172.21.15.37:0/1515810832 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f568c0a04d0 msgr2=0x7f568c1cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f568c0a04d0 0x7f568c1cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 10 monclient: _finish_hunting 0 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 1 monclient: found mon.noname-c 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 1 -- 172.21.15.37:0/1515810832 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5684009a70 con 0x7f568c0a0e90 2024-09-16T07:18:59.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568ad9d700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f568c0a2d50 0x7f568c1ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:18:59.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f5689d9b700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a0e90 0x7f568c1c8330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f567c00e760 tx=0x7f567c00eb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 1 -- start start 2024-09-16T07:18:59.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient: init 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 1 -- 172.21.15.37:0/1515810832 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567c013610 con 0x7f568c0a0e90 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 10 monclient: dump: 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.067 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 1 -- 172.21.15.37:0/1515810832 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567c013770 con 0x7f568c0a0e90 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 1 -- 172.21.15.37:0/1515810832 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567c013ae0 con 0x7f568c0a0e90 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 10 monclient: dump: 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.068 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.069 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.069 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.069 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.069 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.069 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.069 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.069 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567bfff700 4 set_mon_vals no callback set 2024-09-16T07:18:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T07:18:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fed00133470 0x7fed0006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient(hunting): picked mon.noname-c con 0x7fed00133470 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fed00133850 0x7fed0009cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient(hunting): picked mon.noname-a con 0x7fed00133850 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed0009d240 0x7fed0009d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient(hunting): picked mon.noname-b con 0x7fed0009d240 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:59.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0009a1f0 con 0x7fed00133850 2024-09-16T07:18:59.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed000a2f10 con 0x7fed00133470 2024-09-16T07:18:59.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed000a2440 con 0x7fed0009d240 2024-09-16T07:18:59.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:59.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 5 monclient: authenticate success, global_id 4271 2024-09-16T07:18:59.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 10 monclient: _renew_subs 2024-09-16T07:18:59.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 1 -- 172.21.15.37:0/1515810832 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568c06d780 con 0x7f568c0a0e90 2024-09-16T07:18:59.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 10 monclient: _renew_subs 2024-09-16T07:18:59.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 1 -- 172.21.15.37:0/1515810832 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568c070090 con 0x7f568c0a0e90 2024-09-16T07:18:59.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecff59e700 10 monclient(hunting): authenticate will time out at 859.477130s 2024-09-16T07:18:59.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed0009d240 0x7fed0009d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed0009d240 0x7fed0009d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40046/0 (socket says 172.21.15.37:40046) 2024-09-16T07:18:59.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 1 -- 172.21.15.37:0/767853871 learned_addr learned my addr 172.21.15.37:0/767853871 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:59.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 10 monclient(hunting): get_auth_request con 0x7fed0009d240 auth_method 0 2024-09-16T07:18:59.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfe59c700 1 --2- 172.21.15.37:0/767853871 >> v2:172.21.15.143:3300/0 conn(0x7fed00133470 0x7fed0006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfe59c700 10 monclient(hunting): get_auth_request con 0x7fed00133470 auth_method 0 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfe59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfe59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfe59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-09-16T07:18:59.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 1 -- 172.21.15.37:0/767853871 >> v2:172.21.15.143:3300/0 conn(0x7fed00133470 msgr2=0x7fed0006c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:59.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 1 --2- 172.21.15.37:0/767853871 >> v2:172.21.15.143:3300/0 conn(0x7fed00133470 0x7fed0006c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 1 -- 172.21.15.37:0/767853871 >> v2:172.21.15.37:3300/0 conn(0x7fed00133850 msgr2=0x7fed0009cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:59.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 1 --2- 172.21.15.37:0/767853871 >> v2:172.21.15.37:3300/0 conn(0x7fed00133850 0x7fed0009cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 10 monclient: _finish_hunting 0 2024-09-16T07:18:59.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 1 monclient: found mon.noname-b 2024-09-16T07:18:59.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:59.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 1 -- 172.21.15.37:0/767853871 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0009a620 con 0x7fed0009d240 2024-09-16T07:18:59.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fecfed9d700 1 --2- 172.21.15.37:0/767853871 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed0009d240 0x7fed0009d6b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fed000a1400 tx=0x7fecf800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf1b6360c1c82e42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 1 -- 172.21.15.37:0/1515810832 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f567c013c40 con 0x7f568c0a0e90 2024-09-16T07:18:59.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 1 -- 172.21.15.37:0/3401530999 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500052d0 con 0x7fd05c071a90 2024-09-16T07:18:59.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:59.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 10 monclient: dump: 2024-09-16T07:18:59.076 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.076 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 5 monclient: authenticate success, global_id 4268 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 1 -- 172.21.15.37:0/3401530999 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd050005430 con 0x7fd05c071a90 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 1 -- 172.21.15.37:0/3401530999 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500055a0 con 0x7fd05c071a90 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 10 monclient: dump: 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.078 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.079 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.079 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568b59e700 1 -- 172.21.15.37:0/1515810832 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c0a1270 con 0x7f568c0a0e90 2024-09-16T07:18:59.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd059ffb700 4 set_mon_vals no callback set 2024-09-16T07:18:59.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 10 monclient: _renew_subs 2024-09-16T07:18:59.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 1 -- 172.21.15.37:0/3401530999 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c13dbe0 con 0x7fd05c071a90 2024-09-16T07:18:59.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 10 monclient: _renew_subs 2024-09-16T07:18:59.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 1 -- 172.21.15.37:0/3401530999 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c11d050 con 0x7fd05c071a90 2024-09-16T07:18:59.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f566c035600 0x7f566c037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f567affd700 1 -- 172.21.15.37:0/1515810832 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7f567c047770 con 0x7f568c0a0e90 2024-09-16T07:18:59.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568a59c700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f566c035600 0x7f566c037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568a59c700 10 monclient: get_auth_request con 0x7f566c035600 auth_method 0 2024-09-16T07:18:59.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 1 -- 172.21.15.37:0/3401530999 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fd0500038c0 con 0x7fd05c071a90 2024-09-16T07:18:59.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd058ff9700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd048035600 0x7fd048037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7f568a59c700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f566c035600 0x7f566c037ac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5680005930 tx=0x7f5680009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.988+0000 7fd062274700 1 -- 172.21.15.37:0/3401530999 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c071550 con 0x7fd05c071a90 2024-09-16T07:18:59.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fd05bfff700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd048035600 0x7fd048037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fd05bfff700 10 monclient: get_auth_request con 0x7fd048035600 auth_method 0 2024-09-16T07:18:59.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 1 -- 172.21.15.37:0/767853871 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800e070 con 0x7fed0009d240 2024-09-16T07:18:59.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:18:59.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 10 monclient: dump: 2024-09-16T07:18:59.083 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.083 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 1 -- 172.21.15.37:0/767853871 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf800b9c0 con 0x7fed0009d240 2024-09-16T07:18:59.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 1 -- 172.21.15.37:0/767853871 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800bd50 con 0x7fed0009d240 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecfd59a700 10 monclient: dump: 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.085 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.086 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7f7986c0a700 10 monclient: get_monmap_and_config 2024-09-16T07:18:59.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7f7986c0a700 10 monclient: build_initial_monmap 2024-09-16T07:18:59.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7f7986c0a700 10 monclient: monmap: 2024-09-16T07:18:59.086 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:59.086 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.086 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:58.995573+0000 2024-09-16T07:18:59.086 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:58.995573+0000 2024-09-16T07:18:59.087 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:59.087 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.087 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:18:59.087 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:18:59.087 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:18:59.087 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:18:59.087 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:18:59.087 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7f5d037fe700 1 -- 172.21.15.37:0/362898632 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5d14003dc0 con 0x7f5d1c071680 2024-09-16T07:18:59.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7f5d037fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7f5d037fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:59.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecff59e700 5 monclient: authenticate success, global_id 4274 2024-09-16T07:18:59.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecff59e700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:59.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecff59e700 4 set_mon_vals no callback set 2024-09-16T07:18:59.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fd05bfff700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd048035600 0x7fd048037ac0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd05400b370 tx=0x7fd054009000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecff59e700 10 monclient: shutdown 2024-09-16T07:18:59.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecff59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecff59e700 1 -- 172.21.15.37:0/767853871 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed0009d240 msgr2=0x7fed0009d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fecff59e700 1 --2- 172.21.15.37:0/767853871 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed0009d240 0x7fed0009d6b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fed000a1400 tx=0x7fecf800b330 comp rx=0 tx=0).stop 2024-09-16T07:18:59.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.992+0000 7fd058ff9700 1 -- 172.21.15.37:0/3401530999 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7fd050010d90 con 0x7fd05c071a90 2024-09-16T07:18:59.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7f567affd700 1 -- 172.21.15.37:0/1515810832 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f567c01b960 con 0x7f568c0a0e90 2024-09-16T07:18:59.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7f567affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7f567affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:59.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7fecff59e700 1 -- 172.21.15.37:0/767853871 shutdown_connections 2024-09-16T07:18:59.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7fecff59e700 1 --2- 172.21.15.37:0/767853871 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed0009d240 0x7fed0009d6b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7fecff59e700 1 --2- 172.21.15.37:0/767853871 >> v2:172.21.15.37:3300/0 conn(0x7fed00133850 0x7fed0009cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7fecff59e700 1 --2- 172.21.15.37:0/767853871 >> v2:172.21.15.143:3300/0 conn(0x7fed00133470 0x7fed0006c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7fecff59e700 1 -- 172.21.15.37:0/767853871 >> 172.21.15.37:0/767853871 conn(0x7fed000a3c60 msgr2=0x7fed000a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7fdf3cff9700 1 -- 172.21.15.37:0/1140362169 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdf2800f040 con 0x7fdf40071590 2024-09-16T07:18:59.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7fdf3cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:58.996+0000 7fdf3cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:59.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7f7986c0a700 1 Processor -- start 2024-09-16T07:18:59.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7fecff59e700 1 -- 172.21.15.37:0/767853871 shutdown_connections 2024-09-16T07:18:59.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7fecff59e700 1 -- 172.21.15.37:0/767853871 wait complete. 2024-09-16T07:18:59.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7f7986c0a700 1 -- start start 2024-09-16T07:18:59.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7f7986c0a700 10 monclient: init 2024-09-16T07:18:59.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7fd058ff9700 1 -- 172.21.15.37:0/3401530999 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd050017020 con 0x7fd05c071a90 2024-09-16T07:18:59.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7fd058ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7fd058ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7f7986c0a700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7f7986c0a700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7f7986c0a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f79800a0a50 0x7f79800a2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7f7986c0a700 10 monclient(hunting): picked mon.noname-c con 0x7f79800a0a50 addr v2:172.21.15.143:3300/0 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7f7986c0a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79800a2c90 addr v1:172.21.15.37:6789/0 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7f7986c0a700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f79800a30c0 0x7f7980139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.000+0000 7f7986c0a700 10 monclient(hunting): picked mon.noname-a con 0x7f79800a30c0 addr v2:172.21.15.37:3300/0 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798009c570 con 0x7f79800a2c90 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f798009b890 con 0x7f79800a30c0 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7980099be0 con 0x7f79800a0a50 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 10 monclient(hunting): authenticate will time out at 859.492052s 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient: build_initial_monmap 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient: monmap: 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:59.095 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7985407700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f79800a2c90 0x7f7980139810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36650/0 (socket says 172.21.15.37:36650) 2024-09-16T07:18:59.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f79800a30c0 0x7f7980139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f79800a30c0 0x7f7980139f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36548/0 (socket says 172.21.15.37:36548) 2024-09-16T07:18:59.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 1 -- 172.21.15.37:0/198313532 learned_addr learned my addr 172.21.15.37:0/198313532 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:59.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 10 monclient(hunting): get_auth_request con 0x7f79800a30c0 auth_method 0 2024-09-16T07:18:59.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7985c08700 1 --2- 172.21.15.37:0/198313532 >> v2:172.21.15.143:3300/0 conn(0x7f79800a0a50 0x7f79800a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7985c08700 10 monclient(hunting): get_auth_request con 0x7f79800a0a50 auth_method 0 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7985c08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7985c08700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7985c08700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7985c08700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7985c08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7985c08700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 1 Processor -- start 2024-09-16T07:18:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 1 -- start start 2024-09-16T07:18:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient: init 2024-09-16T07:18:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2024-09-16T07:18:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 1 -- 172.21.15.37:0/198313532 >> v2:172.21.15.143:3300/0 conn(0x7f79800a0a50 msgr2=0x7f79800a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 1 --2- 172.21.15.37:0/198313532 >> v2:172.21.15.143:3300/0 conn(0x7f79800a0a50 0x7f79800a2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 1 -- 172.21.15.37:0/198313532 >> v1:172.21.15.37:6789/0 conn(0x7f79800a2c90 legacy=0x7f7980139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:18:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 10 monclient: _finish_hunting 0 2024-09-16T07:18:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 1 monclient: found mon.noname-a 2024-09-16T07:18:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 1 -- 172.21.15.37:0/198313532 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798009c7d0 con 0x7f79800a30c0 2024-09-16T07:18:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986409700 1 --2- 172.21.15.37:0/198313532 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79800a30c0 0x7f7980139f20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f798009c9c0 tx=0x7f797800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64604eae4c42307 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed0009d240 0x7fed00126ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient(hunting): picked mon.noname-a con 0x7fed0009d240 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed00133470 0x7fed00126ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient(hunting): picked mon.noname-c con 0x7fed00133470 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed00133850 0x7fed00127530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient(hunting): picked mon.noname-b con 0x7fed00133850 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed00131de0 con 0x7fed0009d240 2024-09-16T07:18:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed00127da0 con 0x7fed00133850 2024-09-16T07:18:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0012a0e0 con 0x7fed00133470 2024-09-16T07:18:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecff59e700 10 monclient(hunting): authenticate will time out at 859.493632s 2024-09-16T07:18:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 1 -- 172.21.15.37:0/198313532 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797800e070 con 0x7f79800a30c0 2024-09-16T07:18:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 10 monclient: dump: 2024-09-16T07:18:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 5 monclient: authenticate success, global_id 4311 2024-09-16T07:18:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:18:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed0009d240 0x7fed00126ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfdd9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed00133470 0x7fed00126ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed0009d240 0x7fed00126ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36562/0 (socket says 172.21.15.37:36562) 2024-09-16T07:18:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 1 -- 172.21.15.37:0/133201009 learned_addr learned my addr 172.21.15.37:0/133201009 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 10 monclient(hunting): get_auth_request con 0x7fed0009d240 auth_method 0 2024-09-16T07:18:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfdd9b700 10 monclient(hunting): get_auth_request con 0x7fed00133470 auth_method 0 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfed9d700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed00133850 0x7fed00127530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfed9d700 10 monclient(hunting): get_auth_request con 0x7fed00133850 auth_method 0 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 1 -- 172.21.15.37:0/133201009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed00133470 msgr2=0x7fed00126ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed00133470 0x7fed00126ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 1 -- 172.21.15.37:0/133201009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed00133850 msgr2=0x7fed00127530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed00133850 0x7fed00127530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 10 monclient: _finish_hunting 0 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 1 monclient: found mon.noname-a 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 1 -- 172.21.15.37:0/133201009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf8009b30 con 0x7fed0009d240 2024-09-16T07:18:59.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 1 -- 172.21.15.37:0/198313532 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7978003a40 con 0x7f79800a30c0 2024-09-16T07:18:59.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 10 monclient: get_monmap_and_config success 2024-09-16T07:18:59.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 1 -- 172.21.15.37:0/198313532 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978003dd0 con 0x7f79800a30c0 2024-09-16T07:18:59.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 4 set_mon_vals no callback set 2024-09-16T07:18:59.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7984c06700 10 monclient: dump: 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7fecfe59c700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed0009d240 0x7fed00126ab0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fecf000e760 tx=0x7fecf000ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 10 monclient: shutdown 2024-09-16T07:18:59.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 1 -- 172.21.15.37:0/198313532 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79800a30c0 msgr2=0x7f7980139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 1 --2- 172.21.15.37:0/198313532 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79800a30c0 0x7f7980139f20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f798009c9c0 tx=0x7f797800b330 comp rx=0 tx=0).stop 2024-09-16T07:18:59.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 1 -- 172.21.15.37:0/198313532 shutdown_connections 2024-09-16T07:18:59.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 1 --2- 172.21.15.37:0/198313532 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79800a30c0 0x7f7980139f20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 1 --2- 172.21.15.37:0/198313532 >> v2:172.21.15.143:3300/0 conn(0x7f79800a0a50 0x7f79800a2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.004+0000 7f7986c0a700 1 -- 172.21.15.37:0/198313532 >> 172.21.15.37:0/198313532 conn(0x7f79800a3ea0 msgr2=0x7f79800a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 -- 172.21.15.37:0/198313532 shutdown_connections 2024-09-16T07:18:59.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 -- 172.21.15.37:0/198313532 wait complete. 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient: build_initial_monmap 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient: monmap: 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 Processor -- start 2024-09-16T07:18:59.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 -- start start 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient: init 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient: _reopen_session rank -1 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79800a0a50 0x7f79801d6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient(hunting): picked mon.noname-c con 0x7f79800a0a50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79800a2c90 0x7f79801d7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient(hunting): picked mon.noname-b con 0x7f79800a2c90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79800a30c0 0x7f7980137310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient(hunting): picked mon.noname-a con 0x7f79800a30c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798013a6f0 con 0x7f79800a30c0 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7980139a50 con 0x7f79800a2c90 2024-09-16T07:18:59.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79800724c0 con 0x7f79800a0a50 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient(hunting): _renew_subs 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient(hunting): authenticate will time out at 859.498068s 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986409700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79800a30c0 0x7f7980137310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79800a0a50 0x7f79801d6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986409700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79800a30c0 0x7f7980137310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36572/0 (socket says 172.21.15.37:36572) 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79800a0a50 0x7f79801d6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40074/0 (socket says 172.21.15.37:40074) 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986409700 1 -- 172.21.15.37:0/3669755571 learned_addr learned my addr 172.21.15.37:0/3669755571 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986409700 10 monclient(hunting): get_auth_request con 0x7f79800a30c0 auth_method 0 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986409700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986409700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986409700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 10 monclient(hunting): get_auth_request con 0x7f79800a0a50 auth_method 0 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986409700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986409700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986409700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985407700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79800a2c90 0x7f79801d7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985407700 10 monclient(hunting): get_auth_request con 0x7f79800a2c90 auth_method 0 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985407700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985407700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985407700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-09-16T07:18:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 1 -- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79800a2c90 msgr2=0x7f79801d7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79800a2c90 0x7f79801d7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 1 -- 172.21.15.37:0/3669755571 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79800a30c0 msgr2=0x7f7980137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79800a30c0 0x7f7980137310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 10 monclient: _finish_hunting 0 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 1 monclient: found mon.noname-c 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 1 -- 172.21.15.37:0/3669755571 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7978009b30 con 0x7f79800a0a50 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985407700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79800a2c90 0x7f79801d7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7985c08700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79800a0a50 0x7f79801d6d30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f797c00e8d0 tx=0x7f797c00ebe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f79767fc700 1 -- 172.21.15.37:0/3669755571 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c013680 con 0x7f79800a0a50 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f79767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f79767fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f79767fc700 10 monclient: dump: 2024-09-16T07:18:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f79767fc700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f79767fc700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f79767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 5 monclient: authenticate success, global_id 4280 2024-09-16T07:18:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient: _renew_subs 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 -- 172.21.15.37:0/3669755571 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798011a860 con 0x7f79800a0a50 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient: _renew_subs 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.008+0000 7f7986c0a700 1 -- 172.21.15.37:0/3669755571 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7980139760 con 0x7f79800a0a50 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f79767fc700 1 -- 172.21.15.37:0/3669755571 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c0137e0 con 0x7f79800a0a50 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f79767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f79767fc700 1 -- 172.21.15.37:0/3669755571 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c013b70 con 0x7f79800a0a50 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f79767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f79767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f79767fc700 10 monclient: dump: 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f79767fc700 1 -- 172.21.15.37:0/3669755571 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f797c013cd0 con 0x7f79800a0a50 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f79767fc700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f796c0357b0 0x7f796c037c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f79767fc700 1 -- 172.21.15.37:0/3669755571 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7f797c0479c0 con 0x7f79800a0a50 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f7985407700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f796c0357b0 0x7f796c037c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f7985407700 10 monclient: get_auth_request con 0x7f796c0357b0 auth_method 0 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.012+0000 7f7985407700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f796c0357b0 0x7f796c037c70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7970000f60 tx=0x7f7970007e10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7f7977fff700 4 set_mon_vals no callback set 2024-09-16T07:18:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 1 -- 172.21.15.37:0/133201009 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0013610 con 0x7fed0009d240 2024-09-16T07:18:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:18:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 10 monclient: dump: 2024-09-16T07:18:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 10 monclient: _finish_auth 0 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 10 monclient: _check_auth_tickets 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7fecff59e700 5 monclient: authenticate success, global_id 4317 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7fecff59e700 10 monclient: _renew_subs 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7fecff59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7fecff59e700 1 -- 172.21.15.37:0/133201009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed00070710 con 0x7fed0009d240 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7fecff59e700 10 monclient: _renew_subs 2024-09-16T07:18:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7fecff59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7fecff59e700 1 -- 172.21.15.37:0/133201009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed00129b20 con 0x7fed0009d240 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 1 -- 172.21.15.37:0/133201009 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf0013770 con 0x7fed0009d240 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceffff700 4 set_mon_vals no callback set 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 1 -- 172.21.15.37:0/133201009 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0013ae0 con 0x7fed0009d240 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 10 monclient: dump: 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:18:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:18:59.124 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:18:59.124 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:18:59.124 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:18:59.124 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:18:59.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 1 -- 172.21.15.37:0/133201009 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fecf0013c40 con 0x7fed0009d240 2024-09-16T07:18:59.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fece0035650 0x7fece0037b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:18:59.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7feceeffd700 1 -- 172.21.15.37:0/133201009 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7fecf0047630 con 0x7fed0009d240 2024-09-16T07:18:59.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7fecfdd9b700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fece0035650 0x7fece0037b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:18:59.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7fecfdd9b700 10 monclient: get_auth_request con 0x7fece0035650 auth_method 0 2024-09-16T07:18:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.016+0000 7fecfdd9b700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fece0035650 0x7fece0037b10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fed0011e250 tx=0x7fecf40091c0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:18:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.024+0000 7f7986c0a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.024+0000 7f7986c0a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.024+0000 7f7986c0a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.024+0000 7fecff59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.024+0000 7fecff59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.024+0000 7fecff59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.024+0000 7f7986c0a700 1 -- 172.21.15.37:0/3669755571 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7968005100 con 0x7f79800a0a50 2024-09-16T07:18:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.024+0000 7fecff59e700 1 -- 172.21.15.37:0/133201009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece4005100 con 0x7fed0009d240 2024-09-16T07:18:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.028+0000 7feceeffd700 1 -- 172.21.15.37:0/133201009 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fecf0012370 con 0x7fed0009d240 2024-09-16T07:18:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.028+0000 7feceeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.028+0000 7feceeffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.032+0000 7f79767fc700 1 -- 172.21.15.37:0/3669755571 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f797c017080 con 0x7f79800a0a50 2024-09-16T07:18:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.032+0000 7f79767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:18:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.032+0000 7f79767fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:18:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.180+0000 7fd062274700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T07:18:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.180+0000 7fd062274700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T07:18:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.180+0000 7fd062274700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.180+0000 7fd062274700 1 -- 172.21.15.37:0/3401530999 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd05c09f7e0 con 0x7fd05c071a90 2024-09-16T07:18:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.180+0000 7fd058ff9700 1 -- 172.21.15.37:0/3401530999 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd050010b60 con 0x7fd05c071a90 2024-09-16T07:18:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.180+0000 7fd058ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T07:18:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.180+0000 7fd058ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:59.183 INFO:teuthology.orchestra.run.smithi037.stdout:55834574852 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 -- 172.21.15.37:0/3401530999 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd048035600 msgr2=0x7fd048037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd048035600 0x7fd048037ac0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd05400b370 tx=0x7fd054009000 comp rx=0 tx=0).stop 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 10 monclient: shutdown 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 -- 172.21.15.37:0/3401530999 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071a90 msgr2=0x7fd05c135190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071a90 0x7fd05c135190 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd05000c6b0 tx=0x7fd05000c9c0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 -- 172.21.15.37:0/3401530999 shutdown_connections 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd05c09fa00 0x7fd05c11ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd048035600 0x7fd048037ac0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071a90 0x7fd05c135190 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 --2- 172.21.15.37:0/3401530999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c071170 0x7fd05c134c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 -- 172.21.15.37:0/3401530999 >> 172.21.15.37:0/3401530999 conn(0x7fd05c0a04b0 msgr2=0x7fd05c0a2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 -- 172.21.15.37:0/3401530999 shutdown_connections 2024-09-16T07:18:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.184+0000 7fd062274700 1 -- 172.21.15.37:0/3401530999 wait complete. 2024-09-16T07:18:59.196 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.3 2024-09-16T07:18:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T07:18:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T07:18:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 -- 172.21.15.37:0/3669755571 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f79680053b0 con 0x7f79800a0a50 2024-09-16T07:18:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f79767fc700 1 -- 172.21.15.37:0/3669755571 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f797c01bbe0 con 0x7f79800a0a50 2024-09-16T07:18:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f79767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T07:18:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f79767fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:59.208 INFO:teuthology.orchestra.run.smithi037.stdout:55834574853 2024-09-16T07:18:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 -- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f796c0357b0 msgr2=0x7f796c037c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f796c0357b0 0x7f796c037c70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7970000f60 tx=0x7f7970007e10 comp rx=0 tx=0).stop 2024-09-16T07:18:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 10 monclient: shutdown 2024-09-16T07:18:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 -- 172.21.15.37:0/3669755571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79800a0a50 msgr2=0x7f79801d6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79800a0a50 0x7f79801d6d30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f797c00e8d0 tx=0x7f797c00ebe0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 -- 172.21.15.37:0/3669755571 shutdown_connections 2024-09-16T07:18:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79800a30c0 0x7f7980137310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79800a2c90 0x7f79801d7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f796c0357b0 0x7f796c037c70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 --2- 172.21.15.37:0/3669755571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79800a0a50 0x7f79801d6d30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 -- 172.21.15.37:0/3669755571 >> 172.21.15.37:0/3669755571 conn(0x7f79800a3ea0 msgr2=0x7f798009e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 -- 172.21.15.37:0/3669755571 shutdown_connections 2024-09-16T07:18:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.208+0000 7f7986c0a700 1 -- 172.21.15.37:0/3669755571 wait complete. 2024-09-16T07:18:59.221 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-09-16T07:18:59.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.312+0000 7fdf44a67700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T07:18:59.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.312+0000 7fdf44a67700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T07:18:59.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.312+0000 7fdf44a67700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.312+0000 7fdf44a67700 1 -- 172.21.15.37:0/1140362169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdf400008d0 con 0x7fdf40071590 2024-09-16T07:18:59.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.312+0000 7fdf3cff9700 1 -- 172.21.15.37:0/1140362169 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdf28010310 con 0x7fdf40071590 2024-09-16T07:18:59.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.312+0000 7fdf3cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T07:18:59.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.312+0000 7fdf3cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:59.313 INFO:teuthology.orchestra.run.smithi037.stdout:47244640259 2024-09-16T07:18:59.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 -- 172.21.15.37:0/1140362169 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdf20035650 msgr2=0x7fdf20037b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdf20035650 0x7fdf20037b10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf4009ca30 tx=0x7fdf30009040 comp rx=0 tx=0).stop 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 10 monclient: shutdown 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 -- 172.21.15.37:0/1140362169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 msgr2=0x7fdf401ca230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 0x7fdf401ca230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800e790 tx=0x7fdf2800eaa0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 -- 172.21.15.37:0/1140362169 shutdown_connections 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf400a2c80 0x7fdf40121190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdf20035650 0x7fdf20037b10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf400a2360 0x7fdf401ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 --2- 172.21.15.37:0/1140362169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf40071590 0x7fdf401ca230 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 -- 172.21.15.37:0/1140362169 >> 172.21.15.37:0/1140362169 conn(0x7fdf400a3c60 msgr2=0x7fdf4009b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 -- 172.21.15.37:0/1140362169 shutdown_connections 2024-09-16T07:18:59.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.316+0000 7fdf44a67700 1 -- 172.21.15.37:0/1140362169 wait complete. 2024-09-16T07:18:59.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T07:18:59.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T07:18:59.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 -- 172.21.15.37:0/362898632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5d1c123ea0 con 0x7f5d1c071680 2024-09-16T07:18:59.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d037fe700 1 -- 172.21.15.37:0/362898632 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5d14019300 con 0x7f5d1c071680 2024-09-16T07:18:59.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d037fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T07:18:59.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d037fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:59.320 INFO:teuthology.orchestra.run.smithi037.stdout:55834574853 2024-09-16T07:18:59.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 -- 172.21.15.37:0/362898632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5d08035600 msgr2=0x7f5d08037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5d08035600 0x7f5d08037ac0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5d10006f80 tx=0x7f5d1001c040 comp rx=0 tx=0).stop 2024-09-16T07:18:59.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 10 monclient: shutdown 2024-09-16T07:18:59.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 -- 172.21.15.37:0/362898632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c071680 msgr2=0x7f5d1c1ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c071680 0x7f5d1c1ca7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400cc80 tx=0x7f5d1400a340 comp rx=0 tx=0).stop 2024-09-16T07:18:59.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 -- 172.21.15.37:0/362898632 shutdown_connections 2024-09-16T07:18:59.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5d08035600 0x7f5d08037ac0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2a50 0x7f5d1c120b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c071680 0x7f5d1c1ca7a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 --2- 172.21.15.37:0/362898632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1c0707a0 0x7f5d1c1ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 -- 172.21.15.37:0/362898632 >> 172.21.15.37:0/362898632 conn(0x7f5d1c0a3c60 msgr2=0x7f5d1c09b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.320+0000 7f5d208af700 1 -- 172.21.15.37:0/362898632 shutdown_connections 2024-09-16T07:18:59.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.324+0000 7f5d208af700 1 -- 172.21.15.37:0/362898632 wait complete. 2024-09-16T07:18:59.327 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.5 2024-09-16T07:18:59.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.336+0000 7f568b59e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T07:18:59.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.336+0000 7f568b59e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T07:18:59.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.336+0000 7f568b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:18:59.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.336+0000 7f568b59e700 1 -- 172.21.15.37:0/1515810832 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f568c0a2110 con 0x7f568c0a0e90 2024-09-16T07:18:59.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.336+0000 7f567affd700 1 -- 172.21.15.37:0/1515810832 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f567c01b730 con 0x7f568c0a0e90 2024-09-16T07:18:59.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.336+0000 7f567affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T07:18:59.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.336+0000 7f567affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:59.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2024-09-16T07:18:59.338 INFO:teuthology.orchestra.run.smithi037.stdout:51539607555 2024-09-16T07:18:59.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 -- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f566c035600 msgr2=0x7f566c037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f566c035600 0x7f566c037ac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5680005930 tx=0x7f5680009040 comp rx=0 tx=0).stop 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 10 monclient: shutdown 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 -- 172.21.15.37:0/1515810832 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a0e90 msgr2=0x7f568c1c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a0e90 0x7f568c1c8330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f567c00e760 tx=0x7f567c00eb20 comp rx=0 tx=0).stop 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 -- 172.21.15.37:0/1515810832 shutdown_connections 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f568c0a2d50 0x7f568c1ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f566c035600 0x7f566c037ac0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f568c0a0e90 0x7f568c1c8330 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 --2- 172.21.15.37:0/1515810832 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f568c0a04d0 0x7f568c1cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 -- 172.21.15.37:0/1515810832 >> 172.21.15.37:0/1515810832 conn(0x7f568c0a4740 msgr2=0x7f568c0a70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 -- 172.21.15.37:0/1515810832 shutdown_connections 2024-09-16T07:18:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.340+0000 7f568b59e700 1 -- 172.21.15.37:0/1515810832 wait complete. 2024-09-16T07:18:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.344+0000 7fecff59e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T07:18:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.344+0000 7fecff59e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T07:18:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.344+0000 7fecff59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:18:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.344+0000 7fecff59e700 1 -- 172.21.15.37:0/133201009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fece4005680 con 0x7fed0009d240 2024-09-16T07:18:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.344+0000 7feceeffd700 1 -- 172.21.15.37:0/133201009 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fecf0010310 con 0x7fed0009d240 2024-09-16T07:18:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.344+0000 7feceeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T07:18:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.344+0000 7feceeffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:18:59.347 INFO:teuthology.orchestra.run.smithi037.stdout:51539607556 2024-09-16T07:18:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 -- 172.21.15.37:0/133201009 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fece0035650 msgr2=0x7fece0037b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fece0035650 0x7fece0037b10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fed0011e250 tx=0x7fecf40091c0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 10 monclient: shutdown 2024-09-16T07:18:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:18:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 -- 172.21.15.37:0/133201009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed0009d240 msgr2=0x7fed00126ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:18:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed0009d240 0x7fed00126ab0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fecf000e760 tx=0x7fecf000ea70 comp rx=0 tx=0).stop 2024-09-16T07:18:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 -- 172.21.15.37:0/133201009 shutdown_connections 2024-09-16T07:18:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed00133850 0x7fed00127530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fece0035650 0x7fece0037b10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed00133470 0x7fed00126ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 --2- 172.21.15.37:0/133201009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed0009d240 0x7fed00126ab0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:18:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 -- 172.21.15.37:0/133201009 >> 172.21.15.37:0/133201009 conn(0x7fed000a3c60 msgr2=0x7fed0006daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:18:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 -- 172.21.15.37:0/133201009 shutdown_connections 2024-09-16T07:18:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:18:59.348+0000 7fecff59e700 1 -- 172.21.15.37:0/133201009 wait complete. 2024-09-16T07:18:59.351 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.1 2024-09-16T07:18:59.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.4 2024-09-16T07:19:00.196 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T07:19:00.221 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T07:19:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient: get_monmap_and_config 2024-09-16T07:19:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient: monmap: 2024-09-16T07:19:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:00.328395+0000 2024-09-16T07:19:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:00.328395+0000 2024-09-16T07:19:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:00.325 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.334 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T07:19:00.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 1 Processor -- start 2024-09-16T07:19:00.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 1 -- start start 2024-09-16T07:19:00.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient: init 2024-09-16T07:19:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T07:19:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a0e90 0x7fe6540a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient(hunting): picked mon.noname-b con 0x7fe6540a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6540a1270 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6540a2d50 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65409f250 con 0x7fe6540a1270 2024-09-16T07:19:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65409f530 con 0x7fe6540a2d50 2024-09-16T07:19:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65406cf50 con 0x7fe6540a0e90 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe653fff700 10 monclient(hunting): authenticate will time out at 860.814864s 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe6527fc700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe6540a1270 0x7fe6540a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36652/0 (socket says 172.21.15.37:36652) 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe6527fc700 1 -- 172.21.15.37:0/4286194869 learned_addr learned my addr 172.21.15.37:0/4286194869 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 1 --2- 172.21.15.37:0/4286194869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a0e90 0x7fe6540a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 10 monclient(hunting): get_auth_request con 0x7fe6540a0e90 auth_method 0 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe651ffb700 1 -- 172.21.15.37:0/4286194869 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444445349 0 0) 0x7fe65409f250 con 0x7fe6540a1270 2024-09-16T07:19:00.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe651ffb700 10 monclient(hunting): _negotiate 2024-09-16T07:19:00.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe651ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe651ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe651ffb700 10 monclient(hunting): my global_id is 4323 2024-09-16T07:19:00.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe651ffb700 1 -- 172.21.15.37:0/4286194869 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6400035d0 con 0x7fe6540a1270 2024-09-16T07:19:00.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-09-16T07:19:00.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 1 -- 172.21.15.37:0/4286194869 >> v1:172.21.15.143:6789/0 conn(0x7fe6540a2d50 legacy=0x7fe6540a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 1 -- 172.21.15.37:0/4286194869 >> v1:172.21.15.37:6789/0 conn(0x7fe6540a1270 legacy=0x7fe6540a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 1 monclient: found mon.noname-b 2024-09-16T07:19:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 1 -- 172.21.15.37:0/4286194869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65406cbd0 con 0x7fe6540a0e90 2024-09-16T07:19:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe652ffd700 1 --2- 172.21.15.37:0/4286194869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a0e90 0x7fe6540a04d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009880 tx=0x7fe64c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9c37f0c3c89f172 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe651ffb700 1 -- 172.21.15.37:0/4286194869 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c011830 con 0x7fe6540a0e90 2024-09-16T07:19:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.324+0000 7fe651ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 10 monclient: dump: 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.341 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 5 monclient: authenticate success, global_id 4283 2024-09-16T07:19:00.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 1 -- 172.21.15.37:0/4286194869 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c011990 con 0x7fe6540a0e90 2024-09-16T07:19:00.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:00.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 1 -- 172.21.15.37:0/4286194869 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c011d20 con 0x7fe6540a0e90 2024-09-16T07:19:00.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 4 set_mon_vals no callback set 2024-09-16T07:19:00.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe651ffb700 10 monclient: dump: 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient: shutdown 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 -- 172.21.15.37:0/4286194869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a0e90 msgr2=0x7fe6540a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 --2- 172.21.15.37:0/4286194869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a0e90 0x7fe6540a04d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009880 tx=0x7fe64c009b90 comp rx=0 tx=0).stop 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 -- 172.21.15.37:0/4286194869 shutdown_connections 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 --2- 172.21.15.37:0/4286194869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a0e90 0x7fe6540a04d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 -- 172.21.15.37:0/4286194869 >> 172.21.15.37:0/4286194869 conn(0x7fe6540a4740 msgr2=0x7fe6540a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 -- 172.21.15.37:0/4286194869 shutdown_connections 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 -- 172.21.15.37:0/4286194869 wait complete. 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient: monmap: 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:00.345 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 Processor -- start 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 -- start start 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient: init 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6540a0e90 0x7fe6541ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient(hunting): picked mon.noname-b con 0x7fe6540a0e90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6540a1270 0x7fe6541cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient(hunting): picked mon.noname-a con 0x7fe6540a1270 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a2d50 0x7fe654122c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient(hunting): picked mon.noname-c con 0x7fe6540a2d50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe654126230 con 0x7fe6540a1270 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6541254f0 con 0x7fe6540a0e90 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65409dec0 con 0x7fe6540a2d50 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe653fff700 10 monclient(hunting): authenticate will time out at 860.819270s 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6540a1270 0x7fe6541cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6540a1270 0x7fe6541cac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36578/0 (socket says 172.21.15.37:36578) 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 1 -- 172.21.15.37:0/3585835326 learned_addr learned my addr 172.21.15.37:0/3585835326 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 10 monclient(hunting): get_auth_request con 0x7fe6540a1270 auth_method 0 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6537fe700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a2d50 0x7fe654122c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6537fe700 10 monclient(hunting): get_auth_request con 0x7fe6540a2d50 auth_method 0 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6537fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe652ffd700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6540a0e90 0x7fe6541ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe652ffd700 10 monclient(hunting): get_auth_request con 0x7fe6540a0e90 auth_method 0 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe652ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe652ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe652ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 1 -- 172.21.15.37:0/3585835326 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a2d50 msgr2=0x7fe654122c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a2d50 0x7fe654122c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 1 -- 172.21.15.37:0/3585835326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6540a0e90 msgr2=0x7fe6541ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6540a0e90 0x7fe6541ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 1 monclient: found mon.noname-a 2024-09-16T07:19:00.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.328+0000 7fe6527fc700 1 -- 172.21.15.37:0/3585835326 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c0094b0 con 0x7fe6540a1270 2024-09-16T07:19:00.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe6527fc700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6540a1270 0x7fe6541cac10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe63c001200 tx=0x7fe63c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 1 -- 172.21.15.37:0/3585835326 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c005d90 con 0x7fe6540a1270 2024-09-16T07:19:00.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:00.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 10 monclient: dump: 2024-09-16T07:19:00.351 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.351 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.351 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 5 monclient: authenticate success, global_id 4326 2024-09-16T07:19:00.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 1 -- 172.21.15.37:0/3585835326 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c0070d0 con 0x7fe6540a1270 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 1 -- 172.21.15.37:0/3585835326 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c007460 con 0x7fe6540a1270 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 10 monclient: _renew_subs 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 1 -- 172.21.15.37:0/3585835326 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65406bee0 con 0x7fe6540a1270 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe650ff9700 4 set_mon_vals no callback set 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 10 monclient: dump: 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.353 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 10 monclient: _renew_subs 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 1 -- 172.21.15.37:0/3585835326 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6541267f0 con 0x7fe6540a1270 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 1 -- 172.21.15.37:0/3585835326 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fe63c0075c0 con 0x7fe6540a1270 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe653fff700 1 -- 172.21.15.37:0/3585835326 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe654126450 con 0x7fe6540a1270 2024-09-16T07:19:00.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe634035600 0x7fe634037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe64b7fe700 1 -- 172.21.15.37:0/3585835326 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7fe63c046ef0 con 0x7fe6540a1270 2024-09-16T07:19:00.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe652ffd700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe634035600 0x7fe634037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe652ffd700 10 monclient: get_auth_request con 0x7fe634035600 auth_method 0 2024-09-16T07:19:00.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.332+0000 7fe652ffd700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe634035600 0x7fe634037ac0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0090b0 tx=0x7fe64c010000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.356 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T07:19:00.357 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T07:19:00.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.336+0000 7fe64b7fe700 1 -- 172.21.15.37:0/3585835326 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe654126450 con 0x7fe6540a1270 2024-09-16T07:19:00.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.336+0000 7fe64b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.336+0000 7fe64b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:00.364 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T07:19:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient: get_monmap_and_config 2024-09-16T07:19:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient: monmap: 2024-09-16T07:19:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:00.371704+0000 2024-09-16T07:19:00.367 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:00.371704+0000 2024-09-16T07:19:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:00.368 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:00.368 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 1 Processor -- start 2024-09-16T07:19:00.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 1 -- start start 2024-09-16T07:19:00.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient: init 2024-09-16T07:19:00.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T07:19:00.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f82f80a0e90 0x7f82f80a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient(hunting): picked mon.noname-a con 0x7f82f80a0e90 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:00.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82f80a1270 0x7f82f80a2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient(hunting): picked mon.noname-b con 0x7f82f80a1270 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82f80a2b80 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:00.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f80a36e0 con 0x7f82f80a2b80 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f806cf50 con 0x7f82f80a0e90 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f809f810 con 0x7f82f80a1270 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient(hunting): authenticate will time out at 860.858481s 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f82f80a0e90 0x7f82f80a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f82f80a0e90 0x7f82f80a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36582/0 (socket says 172.21.15.37:36582) 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 1 -- 172.21.15.37:0/979192592 learned_addr learned my addr 172.21.15.37:0/979192592 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 10 monclient(hunting): get_auth_request con 0x7f82f80a0e90 auth_method 0 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 1 -- 172.21.15.37:0/979192592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82f80a1270 msgr2=0x7f82f80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 1 --2- 172.21.15.37:0/979192592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82f80a1270 0x7f82f80a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 1 -- 172.21.15.37:0/979192592 >> v1:172.21.15.143:6789/0 conn(0x7f82f80a2b80 legacy=0x7f82f80a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 1 monclient: found mon.noname-a 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 1 -- 172.21.15.37:0/979192592 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f809f250 con 0x7f82f80a0e90 2024-09-16T07:19:00.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fc9f3700 1 --2- 172.21.15.37:0/979192592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a0e90 0x7f82f80a04d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009910 tx=0x7f82f0009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e423484780333751 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 1 -- 172.21.15.37:0/979192592 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f000ba60 con 0x7f82f80a0e90 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 10 monclient: dump: 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.375 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 5 monclient: authenticate success, global_id 4329 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 1 -- 172.21.15.37:0/979192592 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f000bbc0 con 0x7f82f80a0e90 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82fd9f5700 4 set_mon_vals no callback set 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 1 -- 172.21.15.37:0/979192592 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f0009e60 con 0x7f82f80a0e90 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.368+0000 7f82f77fe700 10 monclient: dump: 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 10 monclient: shutdown 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 1 -- 172.21.15.37:0/979192592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a0e90 msgr2=0x7f82f80a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 1 --2- 172.21.15.37:0/979192592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a0e90 0x7f82f80a04d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009910 tx=0x7f82f0009c20 comp rx=0 tx=0).stop 2024-09-16T07:19:00.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 1 -- 172.21.15.37:0/979192592 shutdown_connections 2024-09-16T07:19:00.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 1 --2- 172.21.15.37:0/979192592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82f80a1270 0x7f82f80a2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 1 --2- 172.21.15.37:0/979192592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a0e90 0x7f82f80a04d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 1 -- 172.21.15.37:0/979192592 >> 172.21.15.37:0/979192592 conn(0x7f82f80a4740 msgr2=0x7f82f80a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 1 -- 172.21.15.37:0/979192592 shutdown_connections 2024-09-16T07:19:00.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 1 -- 172.21.15.37:0/979192592 wait complete. 2024-09-16T07:19:00.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 10 monclient: monmap: 2024-09-16T07:19:00.380 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.380 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:00.381 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:00.381 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:00.381 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.381 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:00.381 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:00.381 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:00.381 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:00.381 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 1 Processor -- start 2024-09-16T07:19:00.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 1 -- start start 2024-09-16T07:19:00.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.372+0000 7f82fd9f5700 10 monclient: init 2024-09-16T07:19:00.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:19:00.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f80a0e90 0x7f82f81ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient(hunting): picked mon.noname-b con 0x7f82f80a0e90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:00.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a1270 0x7f82f813c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient(hunting): picked mon.noname-a con 0x7f82f80a1270 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:00.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82f80a2b80 0x7f82f8123230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient(hunting): picked mon.noname-c con 0x7f82f80a2b80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f8126f60 con 0x7f82f80a1270 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f81271b0 con 0x7f82f80a0e90 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f809e0f0 con 0x7f82f80a2b80 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient(hunting): authenticate will time out at 860.864487s 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a1270 0x7f82f813c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd1f4700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82f80a2b80 0x7f82f8123230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a1270 0x7f82f813c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36598/0 (socket says 172.21.15.37:36598) 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 1 -- 172.21.15.37:0/410433747 learned_addr learned my addr 172.21.15.37:0/410433747 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 10 monclient(hunting): get_auth_request con 0x7f82f80a1270 auth_method 0 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd1f4700 10 monclient(hunting): get_auth_request con 0x7f82f80a2b80 auth_method 0 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd1f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd1f4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd1f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fc9f3700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f80a0e90 0x7f82f81ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fc9f3700 10 monclient(hunting): get_auth_request con 0x7f82f80a0e90 auth_method 0 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fc9f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fc9f3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fc9f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd1f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd1f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd1f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fc9f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fc9f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fc9f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 1 -- 172.21.15.37:0/410433747 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82f80a2b80 msgr2=0x7f82f8123230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82f80a2b80 0x7f82f8123230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 1 -- 172.21.15.37:0/410433747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f80a0e90 msgr2=0x7f82f81ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f80a0e90 0x7f82f81ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 1 monclient: found mon.noname-a 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 1 -- 172.21.15.37:0/410433747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f0009570 con 0x7f82f80a1270 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f7fff700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a1270 0x7f82f813c820 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82e800ba80 tx=0x7f82e800bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 1 -- 172.21.15.37:0/410433747 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e800c930 con 0x7f82f80a1270 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 10 monclient: dump: 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 1 -- 172.21.15.37:0/410433747 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e800cab0 con 0x7f82f80a1270 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 5 monclient: authenticate success, global_id 4332 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient: _renew_subs 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 1 -- 172.21.15.37:0/410433747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f806c120 con 0x7f82f80a1270 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f67fc700 4 set_mon_vals no callback set 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient: _renew_subs 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 1 -- 172.21.15.37:0/410433747 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e800ce60 con 0x7f82f80a1270 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82fd9f5700 1 -- 172.21.15.37:0/410433747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f8124de0 con 0x7f82f80a1270 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.376+0000 7f82f57fa700 10 monclient: dump: 2024-09-16T07:19:00.388 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.380+0000 7f82f57fa700 1 -- 172.21.15.37:0/410433747 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f82e801d3b0 con 0x7f82f80a1270 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.380+0000 7f82f57fa700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f82e0035780 0x7f82e0037c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.380+0000 7f82f57fa700 1 -- 172.21.15.37:0/410433747 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7f82e8048430 con 0x7f82f80a1270 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.380+0000 7f82fc9f3700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f82e0035780 0x7f82e0037c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.380+0000 7f82fc9f3700 10 monclient: get_auth_request con 0x7f82e0035780 auth_method 0 2024-09-16T07:19:00.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.380+0000 7f82fd9f5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.380+0000 7f82fd9f5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.380+0000 7f82fd9f5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.380+0000 7f82fd9f5700 1 -- 172.21.15.37:0/410433747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f81272f0 con 0x7f82f80a1270 2024-09-16T07:19:00.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.380+0000 7f82fc9f3700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f82e0035780 0x7f82e0037c40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f82f0000c00 tx=0x7f82f0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.384+0000 7f82f57fa700 1 -- 172.21.15.37:0/410433747 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f82f81272f0 con 0x7f82f80a1270 2024-09-16T07:19:00.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.384+0000 7f82f57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.384+0000 7f82f57fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:00.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.452+0000 7f21ef59e700 10 monclient: get_monmap_and_config 2024-09-16T07:19:00.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.452+0000 7f21ef59e700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.452+0000 7f21ef59e700 10 monclient: monmap: 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:00.458908+0000 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:00.458908+0000 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:00.456 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 Processor -- start 2024-09-16T07:19:00.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 -- start start 2024-09-16T07:19:00.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient: init 2024-09-16T07:19:00.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T07:19:00.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21f00a2360 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:00.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0071590 0x7f21f013a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient(hunting): picked mon.noname-b con 0x7f21f0071590 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21f0070a10 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:00.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f00a33c0 con 0x7f21f0070a10 2024-09-16T07:19:00.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f0099d70 con 0x7f21f00a2360 2024-09-16T07:19:00.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f009a050 con 0x7f21f0071590 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient(hunting): authenticate will time out at 860.945211s 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21eed9d700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f21f0070a10 0x7f21f013c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36668/0 (socket says 172.21.15.37:36668) 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21eed9d700 1 -- 172.21.15.37:0/281380457 learned_addr learned my addr 172.21.15.37:0/281380457 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 1 --2- 172.21.15.37:0/281380457 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0071590 0x7f21f013a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 10 monclient(hunting): get_auth_request con 0x7f21f0071590 auth_method 0 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 1 -- 172.21.15.37:0/281380457 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320989407 0 0) 0x7f21f00a33c0 con 0x7f21f0070a10 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient(hunting): _negotiate 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient(hunting): my global_id is 4335 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 1 -- 172.21.15.37:0/281380457 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d80035d0 con 0x7f21f0070a10 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 1 -- 172.21.15.37:0/281380457 >> v1:172.21.15.143:6789/0 conn(0x7f21f00a2360 legacy=0x7f21f00a2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 1 -- 172.21.15.37:0/281380457 >> v1:172.21.15.37:6789/0 conn(0x7f21f0070a10 legacy=0x7f21f013c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 1 monclient: found mon.noname-b 2024-09-16T07:19:00.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 1 -- 172.21.15.37:0/281380457 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f00a2e50 con 0x7f21f0071590 2024-09-16T07:19:00.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21edd9b700 1 --2- 172.21.15.37:0/281380457 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0071590 0x7f21f013a440 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f21e40098b0 tx=0x7f21e4009bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b89f28be4fde0d22 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 1 -- 172.21.15.37:0/281380457 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4004230 con 0x7f21f0071590 2024-09-16T07:19:00.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:00.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient: dump: 2024-09-16T07:19:00.462 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.462 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.463 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.463 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.463 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.463 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.463 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.463 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.463 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.463 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 1 -- 172.21.15.37:0/281380457 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4004390 con 0x7f21f0071590 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 1 -- 172.21.15.37:0/281380457 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40036c0 con 0x7f21f0071590 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ed59a700 10 monclient: dump: 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.464 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 5 monclient: authenticate success, global_id 4292 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 4 set_mon_vals no callback set 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 10 monclient: shutdown 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 -- 172.21.15.37:0/281380457 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0071590 msgr2=0x7f21f013a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 --2- 172.21.15.37:0/281380457 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0071590 0x7f21f013a440 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f21e40098b0 tx=0x7f21e4009bc0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 -- 172.21.15.37:0/281380457 shutdown_connections 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 --2- 172.21.15.37:0/281380457 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0071590 0x7f21f013a440 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 -- 172.21.15.37:0/281380457 >> 172.21.15.37:0/281380457 conn(0x7f21f00a3c60 msgr2=0x7f21f00a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.456+0000 7f21ef59e700 1 -- 172.21.15.37:0/281380457 shutdown_connections 2024-09-16T07:19:00.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 1 -- 172.21.15.37:0/281380457 wait complete. 2024-09-16T07:19:00.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient: monmap: 2024-09-16T07:19:00.469 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.469 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:00.469 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:00.469 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:00.469 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.469 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 1 Processor -- start 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 1 -- start start 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient: init 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0070a10 0x7f21f01ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient(hunting): picked mon.noname-c con 0x7f21f0070a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21f0071590 0x7f21f01ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient(hunting): picked mon.noname-a con 0x7f21f0071590 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:00.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f00a2360 0x7f21f01cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient(hunting): picked mon.noname-b con 0x7f21f00a2360 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0125af0 con 0x7f21f0071590 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f009f990 con 0x7f21f00a2360 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00724c0 con 0x7f21f0070a10 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ef59e700 10 monclient(hunting): authenticate will time out at 860.950160s 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21f0071590 0x7f21f01ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21f0071590 0x7f21f01ca7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36602/0 (socket says 172.21.15.37:36602) 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 1 -- 172.21.15.37:0/2775399678 learned_addr learned my addr 172.21.15.37:0/2775399678 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 10 monclient(hunting): get_auth_request con 0x7f21f0071590 auth_method 0 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ee59c700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0070a10 0x7f21f01ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21eed9d700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f00a2360 0x7f21f01cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21eed9d700 10 monclient(hunting): get_auth_request con 0x7f21f00a2360 auth_method 0 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21eed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21eed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21eed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ee59c700 10 monclient(hunting): get_auth_request con 0x7f21f0070a10 auth_method 0 2024-09-16T07:19:00.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ee59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ee59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ee59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ee59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ee59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ee59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21eed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21eed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21eed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-09-16T07:19:00.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 1 -- 172.21.15.37:0/2775399678 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0070a10 msgr2=0x7f21f01ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0070a10 0x7f21f01ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 1 -- 172.21.15.37:0/2775399678 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f00a2360 msgr2=0x7f21f01cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f00a2360 0x7f21f01cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 1 monclient: found mon.noname-a 2024-09-16T07:19:00.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21edd9b700 1 -- 172.21.15.37:0/2775399678 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e40094b0 con 0x7f21f0071590 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.460+0000 7f21ee59c700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0070a10 0x7f21f01ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21edd9b700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21f0071590 0x7f21f01ca7f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21e4000c00 tx=0x7f21e4004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 1 -- 172.21.15.37:0/2775399678 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4006ea0 con 0x7f21f0071590 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 10 monclient: dump: 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.475 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 1 -- 172.21.15.37:0/2775399678 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4011630 con 0x7f21f0071590 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ef59e700 5 monclient: authenticate success, global_id 4338 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ef59e700 10 monclient: _renew_subs 2024-09-16T07:19:00.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ef59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ef59e700 1 -- 172.21.15.37:0/2775399678 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f0124e30 con 0x7f21f0071590 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ecd99700 4 set_mon_vals no callback set 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 1 -- 172.21.15.37:0/2775399678 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e40119e0 con 0x7f21f0071590 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ef59e700 10 monclient: _renew_subs 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ef59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ef59e700 1 -- 172.21.15.37:0/2775399678 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f0123300 con 0x7f21f0071590 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 10 monclient: dump: 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.477 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.478 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.478 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.478 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.478 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.478 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.478 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.478 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 1 -- 172.21.15.37:0/2775399678 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f21e4021070 con 0x7f21f0071590 2024-09-16T07:19:00.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f21d0035600 0x7f21d0037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21deffd700 1 -- 172.21.15.37:0/2775399678 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7f21e404a8b0 con 0x7f21f0071590 2024-09-16T07:19:00.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ee59c700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f21d0035600 0x7f21d0037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ee59c700 10 monclient: get_auth_request con 0x7f21d0035600 auth_method 0 2024-09-16T07:19:00.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.464+0000 7f21ee59c700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f21d0035600 0x7f21d0037ac0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f21e0004510 tx=0x7f21e0007ee0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.472+0000 7f21ef59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.472+0000 7f21ef59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.472+0000 7f21ef59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.472+0000 7f21ef59e700 1 -- 172.21.15.37:0/2775399678 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f00a2740 con 0x7f21f0071590 2024-09-16T07:19:00.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7f21deffd700 1 -- 172.21.15.37:0/2775399678 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f21e404e060 con 0x7f21f0071590 2024-09-16T07:19:00.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7f21deffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7f21deffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:00.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient: get_monmap_and_config 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient: monmap: 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:00.485034+0000 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:00.485034+0000 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:00.481 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:00.482 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 1 Processor -- start 2024-09-16T07:19:00.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 1 -- start start 2024-09-16T07:19:00.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient: init 2024-09-16T07:19:00.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T07:19:00.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fddb40a1450 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:00.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fddb40a1f40 0x7fddb40713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient(hunting): picked mon.noname-c con 0x7fddb40a1f40 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:00.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fddb4138440 0x7fddb413a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient(hunting): picked mon.noname-a con 0x7fddb4138440 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:00.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb406f120 con 0x7fddb40a1450 2024-09-16T07:19:00.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb406ef70 con 0x7fddb4138440 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb4070710 con 0x7fddb40a1f40 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddbb0b4700 10 monclient(hunting): authenticate will time out at 860.971179s 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fddb4138440 0x7fddb413a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fddb4138440 0x7fddb413a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36618/0 (socket says 172.21.15.37:36618) 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 1 -- 172.21.15.37:0/3935199171 learned_addr learned my addr 172.21.15.37:0/3935199171 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 10 monclient(hunting): get_auth_request con 0x7fddb4138440 auth_method 0 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddb98b1700 1 --2- 172.21.15.37:0/3935199171 >> v2:172.21.15.143:3300/0 conn(0x7fddb40a1f40 0x7fddb40713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddb98b1700 10 monclient(hunting): get_auth_request con 0x7fddb40a1f40 auth_method 0 2024-09-16T07:19:00.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddb98b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddb98b1700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddb98b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 10 monclient(hunting): handle_auth_done global_id 4341 payload 274 2024-09-16T07:19:00.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 1 -- 172.21.15.37:0/3935199171 >> v2:172.21.15.143:3300/0 conn(0x7fddb40a1f40 msgr2=0x7fddb40713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:00.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 1 --2- 172.21.15.37:0/3935199171 >> v2:172.21.15.143:3300/0 conn(0x7fddb40a1f40 0x7fddb40713d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 1 -- 172.21.15.37:0/3935199171 >> v1:172.21.15.143:6789/0 conn(0x7fddb40a1450 legacy=0x7fddb40a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 1 monclient: found mon.noname-a 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.480+0000 7fddba8b3700 1 -- 172.21.15.37:0/3935199171 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb4070430 con 0x7fddb4138440 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddba8b3700 1 --2- 172.21.15.37:0/3935199171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fddb4138440 0x7fddb413a830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fddb40a00b0 tx=0x7fddac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e8e27249358d05 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddb90b0700 1 -- 172.21.15.37:0/3935199171 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac00e070 con 0x7fddb4138440 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddb90b0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddb90b0700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddb90b0700 10 monclient: dump: 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.488 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddb90b0700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddb90b0700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddb90b0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddbb0b4700 5 monclient: authenticate success, global_id 4341 2024-09-16T07:19:00.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddb90b0700 1 -- 172.21.15.37:0/3935199171 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac00b9c0 con 0x7fddb4138440 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddbb0b4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddb90b0700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddbb0b4700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddbb0b4700 4 set_mon_vals no callback set 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddbb0b4700 10 monclient: shutdown 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddbb0b4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddbb0b4700 1 -- 172.21.15.37:0/3935199171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fddb4138440 msgr2=0x7fddb413a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.484+0000 7fddbb0b4700 1 --2- 172.21.15.37:0/3935199171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fddb4138440 0x7fddb413a830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fddb40a00b0 tx=0x7fddac00b330 comp rx=0 tx=0).stop 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 -- 172.21.15.37:0/3935199171 shutdown_connections 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 --2- 172.21.15.37:0/3935199171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fddb4138440 0x7fddb413a830 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 --2- 172.21.15.37:0/3935199171 >> v2:172.21.15.143:3300/0 conn(0x7fddb40a1f40 0x7fddb40713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 -- 172.21.15.37:0/3935199171 >> 172.21.15.37:0/3935199171 conn(0x7fddb40a2540 msgr2=0x7fddb40a2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 -- 172.21.15.37:0/3935199171 shutdown_connections 2024-09-16T07:19:00.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 -- 172.21.15.37:0/3935199171 wait complete. 2024-09-16T07:19:00.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient: monmap: 2024-09-16T07:19:00.491 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.491 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:00.491 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:00.491 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 Processor -- start 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 -- start start 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient: init 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:19:00.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddb40a1450 0x7fddb409f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient(hunting): picked mon.noname-b con 0x7fddb40a1450 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fddb40a1f40 0x7fddb409aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient(hunting): picked mon.noname-a con 0x7fddb40a1f40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fddb4138440 0x7fddb409b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient(hunting): picked mon.noname-c con 0x7fddb4138440 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb409d320 con 0x7fddb40a1f40 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb409c940 con 0x7fddb40a1450 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb413a4d0 con 0x7fddb4138440 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddbb0b4700 10 monclient(hunting): authenticate will time out at 860.978270s 2024-09-16T07:19:00.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fddb4138440 0x7fddb409b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fddb4138440 0x7fddb409b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40128/0 (socket says 172.21.15.37:40128) 2024-09-16T07:19:00.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 1 -- 172.21.15.37:0/3135841696 learned_addr learned my addr 172.21.15.37:0/3135841696 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 10 monclient(hunting): get_auth_request con 0x7fddb4138440 auth_method 0 2024-09-16T07:19:00.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba0b2700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddb40a1450 0x7fddb409f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba0b2700 10 monclient(hunting): get_auth_request con 0x7fddb40a1450 auth_method 0 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba0b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba0b2700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba0b2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba0b2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba0b2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba0b2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 1 -- 172.21.15.37:0/3135841696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddb40a1450 msgr2=0x7fddb409f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddb40a1450 0x7fddb409f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 1 -- 172.21.15.37:0/3135841696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fddb40a1f40 msgr2=0x7fddb409aeb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fddb40a1f40 0x7fddb409aeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 1 monclient: found mon.noname-c 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 1 -- 172.21.15.37:0/3135841696 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac009b30 con 0x7fddb4138440 2024-09-16T07:19:00.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.488+0000 7fddba8b3700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fddb4138440 0x7fddb409b3f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fddb406ee70 tx=0x7fddac004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 1 -- 172.21.15.37:0/3135841696 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac013d60 con 0x7fddb4138440 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 10 monclient: dump: 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.498 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 1 -- 172.21.15.37:0/3135841696 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac0092e0 con 0x7fddb4138440 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 1 -- 172.21.15.37:0/3135841696 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac013ec0 con 0x7fddb4138440 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 10 monclient: dump: 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.499 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddb88af700 4 set_mon_vals no callback set 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddbb0b4700 5 monclient: authenticate success, global_id 4298 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddbb0b4700 10 monclient: _renew_subs 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddbb0b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddbb0b4700 1 -- 172.21.15.37:0/3135841696 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb4072750 con 0x7fddb4138440 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddbb0b4700 10 monclient: _renew_subs 2024-09-16T07:19:00.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddbb0b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddbb0b4700 1 -- 172.21.15.37:0/3135841696 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb409e6a0 con 0x7fddb4138440 2024-09-16T07:19:00.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 1 -- 172.21.15.37:0/3135841696 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fddac01c640 con 0x7fddb4138440 2024-09-16T07:19:00.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.492+0000 7fddaaffd700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdd98035600 0x7fdd98037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.496+0000 7fddaaffd700 1 -- 172.21.15.37:0/3135841696 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7fddac049280 con 0x7fddb4138440 2024-09-16T07:19:00.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.496+0000 7fddba0b2700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdd98035600 0x7fdd98037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.496+0000 7fddba0b2700 10 monclient: get_auth_request con 0x7fdd98035600 auth_method 0 2024-09-16T07:19:00.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.496+0000 7fddba0b2700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdd98035600 0x7fdd98037ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdda4000f60 tx=0x7fdda4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.496+0000 7fdda8ff9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.496+0000 7fdda8ff9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.496+0000 7fdda8ff9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.496+0000 7fdda8ff9700 1 -- 172.21.15.37:0/3135841696 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb406e6e0 con 0x7fddb4138440 2024-09-16T07:19:00.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.500+0000 7f05fa61b700 10 monclient: get_monmap_and_config 2024-09-16T07:19:00.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.500+0000 7f05fa61b700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.500+0000 7f05fa61b700 10 monclient: monmap: 2024-09-16T07:19:00.504 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.504 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:00.506528+0000 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:00.506528+0000 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.500+0000 7f05fa61b700 1 Processor -- start 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.500+0000 7f05fa61b700 1 -- start start 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.500+0000 7f05fa61b700 10 monclient: init 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05f409aaf0 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f05f409aed0 0x7f05f409b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient(hunting): picked mon.noname-a con 0x7f05f409aed0 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 0x7f05f4139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient(hunting): picked mon.noname-b con 0x7f05f4137640 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f409c7a0 con 0x7f05f409aaf0 2024-09-16T07:19:00.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f409ca80 con 0x7f05f409aed0 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f409cd60 con 0x7f05f4137640 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient(hunting): authenticate will time out at 860.992665s 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 0x7f05f4139a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 0x7f05f4139a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40138/0 (socket says 172.21.15.37:40138) 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 1 -- 172.21.15.37:0/1530118229 learned_addr learned my addr 172.21.15.37:0/1530118229 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 10 monclient(hunting): get_auth_request con 0x7f05f4137640 auth_method 0 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f37fe700 1 --2- 172.21.15.37:0/1530118229 >> v2:172.21.15.37:3300/0 conn(0x7f05f409aed0 0x7f05f409b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f37fe700 10 monclient(hunting): get_auth_request con 0x7f05f409aed0 auth_method 0 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-09-16T07:19:00.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 1 -- 172.21.15.37:0/1530118229 >> v2:172.21.15.37:3300/0 conn(0x7f05f409aed0 msgr2=0x7f05f409b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:00.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 1 --2- 172.21.15.37:0/1530118229 >> v2:172.21.15.37:3300/0 conn(0x7f05f409aed0 0x7f05f409b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 1 -- 172.21.15.37:0/1530118229 >> v1:172.21.15.143:6789/0 conn(0x7f05f409aaf0 legacy=0x7f05f409d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 1 monclient: found mon.noname-b 2024-09-16T07:19:00.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 1 -- 172.21.15.37:0/1530118229 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f409fa70 con 0x7f05f4137640 2024-09-16T07:19:00.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f37fe700 1 --2- 172.21.15.37:0/1530118229 >> v2:172.21.15.37:3300/0 conn(0x7f05f409aed0 0x7f05f409b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:19:00.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f8bb8700 1 --2- 172.21.15.37:0/1530118229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 0x7f05f4139a30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009ed0 tx=0x7f05ec00d540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e55260b6f1515743 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 1 -- 172.21.15.37:0/1530118229 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec00dde0 con 0x7f05f4137640 2024-09-16T07:19:00.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:00.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 10 monclient: dump: 2024-09-16T07:19:00.510 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.510 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.510 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 1 -- 172.21.15.37:0/1530118229 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec011460 con 0x7f05f4137640 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 5 monclient: authenticate success, global_id 4301 2024-09-16T07:19:00.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:00.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 4 set_mon_vals no callback set 2024-09-16T07:19:00.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7fddaaffd700 1 -- 172.21.15.37:0/3135841696 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fddac0269a0 con 0x7fddb4138440 2024-09-16T07:19:00.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7fddaaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7fddaaffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 1 -- 172.21.15.37:0/1530118229 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec011810 con 0x7f05f4137640 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05f2ffd700 10 monclient: dump: 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.513 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 10 monclient: shutdown 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 -- 172.21.15.37:0/1530118229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 msgr2=0x7f05f4139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 --2- 172.21.15.37:0/1530118229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 0x7f05f4139a30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009ed0 tx=0x7f05ec00d540 comp rx=0 tx=0).stop 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f3b81cfc700 10 monclient: get_monmap_and_config 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f3b81cfc700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f3b81cfc700 10 monclient: monmap: 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.514 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.515 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:00.510312+0000 2024-09-16T07:19:00.515 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:00.510312+0000 2024-09-16T07:19:00.515 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.515 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.515 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:00.515 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:00.515 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:00.515 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:00.515 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:00.515 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 -- 172.21.15.37:0/1530118229 shutdown_connections 2024-09-16T07:19:00.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 --2- 172.21.15.37:0/1530118229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 0x7f05f4139a30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 --2- 172.21.15.37:0/1530118229 >> v2:172.21.15.37:3300/0 conn(0x7f05f409aed0 0x7f05f409b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f05fa61b700 1 -- 172.21.15.37:0/1530118229 >> 172.21.15.37:0/1530118229 conn(0x7f05f40a04b0 msgr2=0x7f05f40a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.504+0000 7f3b81cfc700 1 Processor -- start 2024-09-16T07:19:00.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 1 -- 172.21.15.37:0/1530118229 shutdown_connections 2024-09-16T07:19:00.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 1 -- start start 2024-09-16T07:19:00.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient: init 2024-09-16T07:19:00.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T07:19:00.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3b7c0a2360 0x7f3b7c0a2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient(hunting): picked mon.noname-a con 0x7f3b7c0a2360 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:00.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 0x7f3b7c135120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient(hunting): picked mon.noname-b con 0x7f3b7c0a2c80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 1 -- 172.21.15.37:0/1530118229 wait complete. 2024-09-16T07:19:00.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3b7c071590 0x7f3b7c135660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient(hunting): picked mon.noname-c con 0x7f3b7c071590 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:00.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b7c0a33c0 con 0x7f3b7c0a2360 2024-09-16T07:19:00.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b7c099d70 con 0x7f3b7c071590 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c09a050 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient(hunting): authenticate will time out at 860.996589s 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b80cfa700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3b7c0a2360 0x7f3b7c0a2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b80cfa700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3b7c0a2360 0x7f3b7c0a2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36624/0 (socket says 172.21.15.37:36624) 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b80cfa700 1 -- 172.21.15.37:0/2735034667 learned_addr learned my addr 172.21.15.37:0/2735034667 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b80cfa700 10 monclient(hunting): get_auth_request con 0x7f3b7c0a2360 auth_method 0 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b80cfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b80cfa700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b80cfa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 1 --2- 172.21.15.37:0/2735034667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 0x7f3b7c135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 10 monclient(hunting): get_auth_request con 0x7f3b7c0a2c80 auth_method 0 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b80cfa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b80cfa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b80cfa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b814fb700 1 --2- 172.21.15.37:0/2735034667 >> v2:172.21.15.143:3300/0 conn(0x7f3b7c071590 0x7f3b7c135660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b814fb700 10 monclient(hunting): get_auth_request con 0x7f3b7c071590 auth_method 0 2024-09-16T07:19:00.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b814fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b814fb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b814fb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 1 -- 172.21.15.37:0/2735034667 >> v2:172.21.15.143:3300/0 conn(0x7f3b7c071590 msgr2=0x7f3b7c135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 1 --2- 172.21.15.37:0/2735034667 >> v2:172.21.15.143:3300/0 conn(0x7f3b7c071590 0x7f3b7c135660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 1 -- 172.21.15.37:0/2735034667 >> v2:172.21.15.37:3300/0 conn(0x7f3b7c0a2360 msgr2=0x7f3b7c0a2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 1 --2- 172.21.15.37:0/2735034667 >> v2:172.21.15.37:3300/0 conn(0x7f3b7c0a2360 0x7f3b7c0a2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 1 monclient: found mon.noname-b 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 1 -- 172.21.15.37:0/2735034667 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7c0a1fe0 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7bfff700 1 --2- 172.21.15.37:0/2735034667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 0x7f3b7c135120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3b70009850 tx=0x7f3b70009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf9a45a9d440f2de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7b7fe700 1 -- 172.21.15.37:0/2735034667 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b70011830 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7b7fe700 10 monclient: dump: 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.523 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7b7fe700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7b7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 5 monclient: authenticate success, global_id 4304 2024-09-16T07:19:00.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:00.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7b7fe700 1 -- 172.21.15.37:0/2735034667 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b70011990 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b7b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:00.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 4 set_mon_vals no callback set 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient: monmap: 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:00.525 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 10 monclient: shutdown 2024-09-16T07:19:00.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 1 -- 172.21.15.37:0/2735034667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 msgr2=0x7f3b7c135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f3b81cfc700 1 --2- 172.21.15.37:0/2735034667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 0x7f3b7c135120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3b70009850 tx=0x7f3b70009b60 comp rx=0 tx=0).stop 2024-09-16T07:19:00.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 1 Processor -- start 2024-09-16T07:19:00.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 1 -- start start 2024-09-16T07:19:00.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient: init 2024-09-16T07:19:00.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:19:00.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05f409aaf0 0x7f05f412cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient(hunting): picked mon.noname-a con 0x7f05f409aaf0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f409aed0 0x7f05f411a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient(hunting): picked mon.noname-b con 0x7f05f409aed0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 0x7f05f411a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient(hunting): picked mon.noname-c con 0x7f05f4137640 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f40726f0 con 0x7f05f409aaf0 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f411bba0 con 0x7f05f409aed0 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f411b560 con 0x7f05f4137640 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05fa61b700 10 monclient(hunting): authenticate will time out at 860.999730s 2024-09-16T07:19:00.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f3fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05f409aaf0 0x7f05f412cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f3fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05f409aaf0 0x7f05f412cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36638/0 (socket says 172.21.15.37:36638) 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f3fff700 1 -- 172.21.15.37:0/1282411622 learned_addr learned my addr 172.21.15.37:0/1282411622 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f3fff700 10 monclient(hunting): get_auth_request con 0x7f05f409aaf0 auth_method 0 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f8bb8700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 0x7f05f411a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.508+0000 7f05f8bb8700 10 monclient(hunting): get_auth_request con 0x7f05f4137640 auth_method 0 2024-09-16T07:19:00.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f8bb8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f8bb8700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f8bb8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f37fe700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f409aed0 0x7f05f411a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f37fe700 10 monclient(hunting): get_auth_request con 0x7f05f409aed0 auth_method 0 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f8bb8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f8bb8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f8bb8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 1 -- 172.21.15.37:0/1282411622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 msgr2=0x7f05f411a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 0x7f05f411a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 1 -- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f409aed0 msgr2=0x7f05f411a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f409aed0 0x7f05f411a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 1 monclient: found mon.noname-a 2024-09-16T07:19:00.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 1 -- 172.21.15.37:0/1282411622 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e80094b0 con 0x7f05f409aaf0 2024-09-16T07:19:00.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f37fe700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f409aed0 0x7f05f411a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:19:00.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.512+0000 7f05f3fff700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05f409aaf0 0x7f05f412cd40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f05e400c500 tx=0x7f05e400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 -- 172.21.15.37:0/2735034667 shutdown_connections 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 --2- 172.21.15.37:0/2735034667 >> v2:172.21.15.143:3300/0 conn(0x7f3b7c071590 0x7f3b7c135660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 --2- 172.21.15.37:0/2735034667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 0x7f3b7c135120 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 --2- 172.21.15.37:0/2735034667 >> v2:172.21.15.37:3300/0 conn(0x7f3b7c0a2360 0x7f3b7c0a2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 -- 172.21.15.37:0/2735034667 >> 172.21.15.37:0/2735034667 conn(0x7f3b7c0a3c60 msgr2=0x7f3b7c0a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 -- 172.21.15.37:0/2735034667 shutdown_connections 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 -- 172.21.15.37:0/2735034667 wait complete. 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient: monmap: 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:00.536 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 Processor -- start 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 -- start start 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient: init 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b7c071590 0x7f3b7c1ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient(hunting): picked mon.noname-a con 0x7f3b7c071590 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b7c0a2360 0x7f3b7c133640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient(hunting): picked mon.noname-b con 0x7f3b7c0a2360 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 0x7f3b7c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient(hunting): picked mon.noname-c con 0x7f3b7c0a2c80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c124b80 con 0x7f3b7c071590 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c1ca6c0 con 0x7f3b7c0a2360 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7c123180 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b81cfc700 10 monclient(hunting): authenticate will time out at 861.005645s 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b7c071590 0x7f3b7c1ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b7c071590 0x7f3b7c1ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36646/0 (socket says 172.21.15.37:36646) 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 1 -- 172.21.15.37:0/466148246 learned_addr learned my addr 172.21.15.37:0/466148246 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 10 monclient(hunting): get_auth_request con 0x7f3b7c071590 auth_method 0 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 0x7f3b7c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 10 monclient(hunting): get_auth_request con 0x7f3b7c0a2c80 auth_method 0 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b7bfff700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b7c0a2360 0x7f3b7c133640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b7bfff700 10 monclient(hunting): get_auth_request con 0x7f3b7c0a2360 auth_method 0 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 1 -- 172.21.15.37:0/466148246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b7c0a2360 msgr2=0x7f3b7c133640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b7c0a2360 0x7f3b7c133640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 1 -- 172.21.15.37:0/466148246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b7c071590 msgr2=0x7f3b7c1ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b7c071590 0x7f3b7c1ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 1 monclient: found mon.noname-c 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 1 -- 172.21.15.37:0/466148246 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b700094b0 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b80cfa700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b7c071590 0x7f3b7c1ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:19:00.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.516+0000 7f3b814fb700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 0x7f3b7c133b80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3b74007d90 tx=0x7f3b7400e3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 1 -- 172.21.15.37:0/1282411622 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e40052d0 con 0x7f05f409aaf0 2024-09-16T07:19:00.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 1 -- 172.21.15.37:0/466148246 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74018070 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 10 monclient: dump: 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 10 monclient: dump: 2024-09-16T07:19:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 1 -- 172.21.15.37:0/1282411622 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e4005430 con 0x7f05f409aaf0 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 1 -- 172.21.15.37:0/1282411622 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e40055a0 con 0x7f05f409aaf0 2024-09-16T07:19:00.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 1 -- 172.21.15.37:0/466148246 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7400eb30 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:00.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 10 monclient: dump: 2024-09-16T07:19:00.546 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.546 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.547 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f27fc700 4 set_mon_vals no callback set 2024-09-16T07:19:00.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b7affd700 4 set_mon_vals no callback set 2024-09-16T07:19:00.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 5 monclient: authenticate success, global_id 4310 2024-09-16T07:19:00.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 10 monclient: _renew_subs 2024-09-16T07:19:00.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 1 -- 172.21.15.37:0/466148246 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b7c09e810 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 10 monclient: _renew_subs 2024-09-16T07:19:00.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 1 -- 172.21.15.37:0/466148246 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b7c124d60 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05fa61b700 5 monclient: authenticate success, global_id 4350 2024-09-16T07:19:00.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05fa61b700 10 monclient: _renew_subs 2024-09-16T07:19:00.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05fa61b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05fa61b700 1 -- 172.21.15.37:0/1282411622 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ec009b30 con 0x7f05f409aaf0 2024-09-16T07:19:00.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05fa61b700 10 monclient: _renew_subs 2024-09-16T07:19:00.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05fa61b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05fa61b700 1 -- 172.21.15.37:0/1282411622 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f412fdb0 con 0x7f05f409aaf0 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 1 -- 172.21.15.37:0/466148246 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74003a40 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 10 monclient: dump: 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.550 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.551 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.551 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 1 -- 172.21.15.37:0/1282411622 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f05e40036a0 con 0x7f05f409aaf0 2024-09-16T07:19:00.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 1 -- 172.21.15.37:0/466148246 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f3b74014420 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f05e0035650 0x7f05e0037b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f0ff9700 1 -- 172.21.15.37:0/1282411622 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7f05e4046e10 con 0x7f05f409aaf0 2024-09-16T07:19:00.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3b640357f0 0x7f3b64037cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b797fa700 1 -- 172.21.15.37:0/466148246 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4402+0+0 (secure 0 0 0) 0x7f3b74047ac0 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b81cfc700 1 -- 172.21.15.37:0/466148246 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b58005100 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f37fe700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f05e0035650 0x7f05e0037b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b80cfa700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3b640357f0 0x7f3b64037cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f37fe700 10 monclient: get_auth_request con 0x7f05e0035650 auth_method 0 2024-09-16T07:19:00.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b80cfa700 10 monclient: get_auth_request con 0x7f3b640357f0 auth_method 0 2024-09-16T07:19:00.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f05f37fe700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f05e0035650 0x7f05e0037b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f05f4071990 tx=0x7f05e8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.524+0000 7f3b80cfa700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3b640357f0 0x7f3b64037cb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c006fd0 tx=0x7f3b6c0092e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.528+0000 7f05fa61b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.528+0000 7f05fa61b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.528+0000 7f05fa61b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.528+0000 7f05fa61b700 1 -- 172.21.15.37:0/1282411622 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f4136260 con 0x7f05f409aaf0 2024-09-16T07:19:00.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.532+0000 7f3b797fa700 1 -- 172.21.15.37:0/466148246 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3b740128d0 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.532+0000 7f3b797fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.532+0000 7f3b797fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:00.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.536+0000 7f05f0ff9700 1 -- 172.21.15.37:0/1282411622 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f05e40106a0 con 0x7f05f409aaf0 2024-09-16T07:19:00.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.536+0000 7f05f0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.536+0000 7f05f0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:00.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T07:19:00.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T07:19:00.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 -- 172.21.15.37:0/410433747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f82f81272f0 con 0x7f82f80a1270 2024-09-16T07:19:00.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82f57fa700 1 -- 172.21.15.37:0/410433747 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f82f81272f0 con 0x7f82f80a1270 2024-09-16T07:19:00.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82f57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T07:19:00.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82f57fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:00.581 INFO:teuthology.orchestra.run.smithi037.stdout:55834574854 2024-09-16T07:19:00.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 -- 172.21.15.37:0/410433747 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f82e0035780 msgr2=0x7f82e0037c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f82e0035780 0x7f82e0037c40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f82f0000c00 tx=0x7f82f0008030 comp rx=0 tx=0).stop 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 10 monclient: shutdown 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 -- 172.21.15.37:0/410433747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a1270 msgr2=0x7f82f813c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a1270 0x7f82f813c820 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f82e800ba80 tx=0x7f82e800bd90 comp rx=0 tx=0).stop 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 -- 172.21.15.37:0/410433747 shutdown_connections 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f82e0035780 0x7f82e0037c40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82f80a2b80 0x7f82f8123230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82f80a1270 0x7f82f813c820 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 --2- 172.21.15.37:0/410433747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f80a0e90 0x7f82f81ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 -- 172.21.15.37:0/410433747 >> 172.21.15.37:0/410433747 conn(0x7f82f80a4740 msgr2=0x7f82f80a70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 -- 172.21.15.37:0/410433747 shutdown_connections 2024-09-16T07:19:00.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.580+0000 7f82fd9f5700 1 -- 172.21.15.37:0/410433747 wait complete. 2024-09-16T07:19:00.598 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-09-16T07:19:00.598 DEBUG:teuthology.parallel:result is None 2024-09-16T07:19:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.644+0000 7fe653fff700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T07:19:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.644+0000 7fe653fff700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T07:19:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.644+0000 7fe653fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.644+0000 7fe653fff700 1 -- 172.21.15.37:0/3585835326 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe654126450 con 0x7fe6540a1270 2024-09-16T07:19:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.644+0000 7fe64b7fe700 1 -- 172.21.15.37:0/3585835326 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe654126450 con 0x7fe6540a1270 2024-09-16T07:19:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.644+0000 7fe64b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T07:19:00.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.644+0000 7fe64b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:00.645 INFO:teuthology.orchestra.run.smithi037.stdout:55834574853 2024-09-16T07:19:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 -- 172.21.15.37:0/3585835326 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe634035600 msgr2=0x7fe634037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe634035600 0x7fe634037ac0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0090b0 tx=0x7fe64c010000 comp rx=0 tx=0).stop 2024-09-16T07:19:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 10 monclient: shutdown 2024-09-16T07:19:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 -- 172.21.15.37:0/3585835326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6540a1270 msgr2=0x7fe6541cac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6540a1270 0x7fe6541cac10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe63c001200 tx=0x7fe63c00c9e0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 -- 172.21.15.37:0/3585835326 shutdown_connections 2024-09-16T07:19:00.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe634035600 0x7fe634037ac0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe6540a2d50 0x7fe654122c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6540a1270 0x7fe6541cac10 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe63c001200 tx=0x7fe63c00c9e0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 --2- 172.21.15.37:0/3585835326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6540a0e90 0x7fe6541ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 -- 172.21.15.37:0/3585835326 >> 172.21.15.37:0/3585835326 conn(0x7fe6540a4740 msgr2=0x7fe6540a70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 -- 172.21.15.37:0/3585835326 shutdown_connections 2024-09-16T07:19:00.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.648+0000 7fe653fff700 1 -- 172.21.15.37:0/3585835326 wait complete. 2024-09-16T07:19:00.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2024-09-16T07:19:00.658 DEBUG:teuthology.parallel:result is None 2024-09-16T07:19:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.708+0000 7f05fa61b700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T07:19:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.708+0000 7f05fa61b700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T07:19:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.708+0000 7f05fa61b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.708+0000 7f05fa61b700 1 -- 172.21.15.37:0/1282411622 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f05f411bce0 con 0x7f05f409aaf0 2024-09-16T07:19:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.708+0000 7f05f0ff9700 1 -- 172.21.15.37:0/1282411622 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f05f411bce0 con 0x7f05f409aaf0 2024-09-16T07:19:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.708+0000 7f05f0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T07:19:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.708+0000 7f05f0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:00.711 INFO:teuthology.orchestra.run.smithi037.stdout:51539607556 2024-09-16T07:19:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 -- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f05e0035650 msgr2=0x7f05e0037b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f05e0035650 0x7f05e0037b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f05f4071990 tx=0x7f05e8008040 comp rx=0 tx=0).stop 2024-09-16T07:19:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 10 monclient: shutdown 2024-09-16T07:19:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 -- 172.21.15.37:0/1282411622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05f409aaf0 msgr2=0x7f05f412cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05f409aaf0 0x7f05f412cd40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f05e400c500 tx=0x7f05e400c810 comp rx=0 tx=0).stop 2024-09-16T07:19:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 -- 172.21.15.37:0/1282411622 shutdown_connections 2024-09-16T07:19:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f05e0035650 0x7f05e0037b10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05f4137640 0x7f05f411a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f409aed0 0x7f05f411a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 --2- 172.21.15.37:0/1282411622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05f409aaf0 0x7f05f412cd40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 -- 172.21.15.37:0/1282411622 >> 172.21.15.37:0/1282411622 conn(0x7f05f40a04b0 msgr2=0x7f05f41325c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 -- 172.21.15.37:0/1282411622 shutdown_connections 2024-09-16T07:19:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.712+0000 7f05fa61b700 1 -- 172.21.15.37:0/1282411622 wait complete. 2024-09-16T07:19:00.723 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.1 2024-09-16T07:19:00.724 DEBUG:teuthology.parallel:result is None 2024-09-16T07:19:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.732+0000 7f21ef59e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T07:19:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.732+0000 7f21ef59e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T07:19:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.732+0000 7f21ef59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.732+0000 7f21ef59e700 1 -- 172.21.15.37:0/2775399678 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f21f00696e0 con 0x7f21f0071590 2024-09-16T07:19:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.732+0000 7f21deffd700 1 -- 172.21.15.37:0/2775399678 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f21e4018350 con 0x7f21f0071590 2024-09-16T07:19:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.732+0000 7f21deffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T07:19:00.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.732+0000 7f21deffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:00.733 INFO:teuthology.orchestra.run.smithi037.stdout:47244640261 2024-09-16T07:19:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 -- 172.21.15.37:0/2775399678 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f21d0035600 msgr2=0x7f21d0037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f21d0035600 0x7f21d0037ac0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f21e0004510 tx=0x7f21e0007ee0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 10 monclient: shutdown 2024-09-16T07:19:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 -- 172.21.15.37:0/2775399678 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21f0071590 msgr2=0x7f21f01ca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21f0071590 0x7f21f01ca7f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f21e4000c00 tx=0x7f21e4004770 comp rx=0 tx=0).stop 2024-09-16T07:19:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 -- 172.21.15.37:0/2775399678 shutdown_connections 2024-09-16T07:19:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21f00a2360 0x7f21f01cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21f0071590 0x7f21f01ca7f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f21d0035600 0x7f21d0037ac0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 --2- 172.21.15.37:0/2775399678 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21f0070a10 0x7f21f01ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 -- 172.21.15.37:0/2775399678 >> 172.21.15.37:0/2775399678 conn(0x7f21f00a3c60 msgr2=0x7f21f011bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 -- 172.21.15.37:0/2775399678 shutdown_connections 2024-09-16T07:19:00.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.736+0000 7f21ef59e700 1 -- 172.21.15.37:0/2775399678 wait complete. 2024-09-16T07:19:00.745 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640261 for osd.5 2024-09-16T07:19:00.746 DEBUG:teuthology.parallel:result is None 2024-09-16T07:19:00.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.792+0000 7fdda8ff9700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T07:19:00.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.792+0000 7fdda8ff9700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T07:19:00.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.792+0000 7fdda8ff9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.792+0000 7fdda8ff9700 1 -- 172.21.15.37:0/3135841696 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fddb40713d0 con 0x7fddb4138440 2024-09-16T07:19:00.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.792+0000 7fddaaffd700 1 -- 172.21.15.37:0/3135841696 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fddac035090 con 0x7fddb4138440 2024-09-16T07:19:00.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.792+0000 7fddaaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T07:19:00.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.792+0000 7fddaaffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:00.795 INFO:teuthology.orchestra.run.smithi037.stdout:55834574854 2024-09-16T07:19:00.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 -- 172.21.15.37:0/3135841696 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdd98035600 msgr2=0x7fdd98037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdd98035600 0x7fdd98037ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdda4000f60 tx=0x7fdda4009040 comp rx=0 tx=0).stop 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 10 monclient: shutdown 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 -- 172.21.15.37:0/3135841696 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fddb4138440 msgr2=0x7fddb409b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fddb4138440 0x7fddb409b3f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fddb406ee70 tx=0x7fddac004af0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 -- 172.21.15.37:0/3135841696 shutdown_connections 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fdd98035600 0x7fdd98037ac0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fddb4138440 0x7fddb409b3f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fddb40a1f40 0x7fddb409aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 --2- 172.21.15.37:0/3135841696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddb40a1450 0x7fddb409f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 -- 172.21.15.37:0/3135841696 >> 172.21.15.37:0/3135841696 conn(0x7fddb40a2540 msgr2=0x7fddb40a47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 -- 172.21.15.37:0/3135841696 shutdown_connections 2024-09-16T07:19:00.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.796+0000 7fdda8ff9700 1 -- 172.21.15.37:0/3135841696 wait complete. 2024-09-16T07:19:00.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T07:19:00.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T07:19:00.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 -- 172.21.15.37:0/466148246 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3b580053b0 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b797fa700 1 -- 172.21.15.37:0/466148246 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3b740122e0 con 0x7f3b7c0a2c80 2024-09-16T07:19:00.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b797fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T07:19:00.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b797fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:00.804 INFO:teuthology.orchestra.run.smithi037.stdout:51539607557 2024-09-16T07:19:00.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 -- 172.21.15.37:0/466148246 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3b640357f0 msgr2=0x7f3b64037cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3b640357f0 0x7f3b64037cb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c006fd0 tx=0x7f3b6c0092e0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 10 monclient: shutdown 2024-09-16T07:19:00.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 -- 172.21.15.37:0/466148246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 msgr2=0x7f3b7c133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 0x7f3b7c133b80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3b74007d90 tx=0x7f3b7400e3c0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 -- 172.21.15.37:0/466148246 shutdown_connections 2024-09-16T07:19:00.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3b640357f0 0x7f3b64037cb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b7c0a2c80 0x7f3b7c133b80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b7c0a2360 0x7f3b7c133640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 --2- 172.21.15.37:0/466148246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b7c071590 0x7f3b7c1ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 -- 172.21.15.37:0/466148246 >> 172.21.15.37:0/466148246 conn(0x7f3b7c0a3c60 msgr2=0x7f3b7c09b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 -- 172.21.15.37:0/466148246 shutdown_connections 2024-09-16T07:19:00.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.804+0000 7f3b81cfc700 1 -- 172.21.15.37:0/466148246 wait complete. 2024-09-16T07:19:00.808 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-09-16T07:19:00.808 DEBUG:teuthology.parallel:result is None 2024-09-16T07:19:00.817 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.4 2024-09-16T07:19:00.817 DEBUG:teuthology.parallel:result is None 2024-09-16T07:19:00.817 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T07:19:00.818 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:19:00.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.920+0000 7ff22ffff700 10 monclient: get_monmap_and_config 2024-09-16T07:19:00.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.920+0000 7ff22ffff700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.920+0000 7ff22ffff700 10 monclient: monmap: 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:00.926952+0000 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:00.926952+0000 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:00.923 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 1 Processor -- start 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 1 -- start start 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient: init 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff2300a2750 0x7ff2300a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient(hunting): picked mon.noname-c con 0x7ff2300a2750 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2300a3070 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff230133730 0x7ff23013b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient(hunting): picked mon.noname-b con 0x7ff230133730 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff230099be0 con 0x7ff2300a3070 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23009d840 con 0x7ff2300a2750 2024-09-16T07:19:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23009de40 con 0x7ff230133730 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient(hunting): authenticate will time out at 861.412806s 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff230133730 0x7ff23013b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff2267fc700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff2300a3070 0x7ff230133020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53374/0 (socket says 172.21.15.37:53374) 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff230133730 0x7ff23013b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37892/0 (socket says 172.21.15.37:37892) 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff2267fc700 1 -- 172.21.15.37:0/1376750276 learned_addr learned my addr 172.21.15.37:0/1376750276 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22effd700 1 --2- 172.21.15.37:0/1376750276 >> v2:172.21.15.143:3300/0 conn(0x7ff2300a2750 0x7ff2300a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 10 monclient(hunting): get_auth_request con 0x7ff230133730 auth_method 0 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22effd700 10 monclient(hunting): get_auth_request con 0x7ff2300a2750 auth_method 0 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 1 -- 172.21.15.37:0/1376750276 >> v2:172.21.15.143:3300/0 conn(0x7ff2300a2750 msgr2=0x7ff2300a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 1 --2- 172.21.15.37:0/1376750276 >> v2:172.21.15.143:3300/0 conn(0x7ff2300a2750 0x7ff2300a2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 1 -- 172.21.15.37:0/1376750276 >> v1:172.21.15.37:6789/0 conn(0x7ff2300a3070 legacy=0x7ff230133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 1 monclient: found mon.noname-b 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 1 -- 172.21.15.37:0/1376750276 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23009d530 con 0x7ff230133730 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22f7fe700 1 --2- 172.21.15.37:0/1376750276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff230133730 0x7ff23013b310 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff228009d90 tx=0x7ff22800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17c08d34e5eb4a56 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 1 -- 172.21.15.37:0/1376750276 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22800e070 con 0x7ff230133730 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 10 monclient: dump: 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 5 monclient: authenticate success, global_id 4313 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 1 -- 172.21.15.37:0/1376750276 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22800bb20 con 0x7ff230133730 2024-09-16T07:19:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22ffff700 4 set_mon_vals no callback set 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 1 -- 172.21.15.37:0/1376750276 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228012430 con 0x7ff230133730 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.924+0000 7ff22e7fc700 10 monclient: dump: 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient: shutdown 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 -- 172.21.15.37:0/1376750276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff230133730 msgr2=0x7ff23013b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 --2- 172.21.15.37:0/1376750276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff230133730 0x7ff23013b310 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff228009d90 tx=0x7ff22800b330 comp rx=0 tx=0).stop 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 -- 172.21.15.37:0/1376750276 shutdown_connections 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 --2- 172.21.15.37:0/1376750276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff230133730 0x7ff23013b310 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 --2- 172.21.15.37:0/1376750276 >> v2:172.21.15.143:3300/0 conn(0x7ff2300a2750 0x7ff2300a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 -- 172.21.15.37:0/1376750276 >> 172.21.15.37:0/1376750276 conn(0x7ff2300a3ea0 msgr2=0x7ff2300a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 -- 172.21.15.37:0/1376750276 shutdown_connections 2024-09-16T07:19:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 -- 172.21.15.37:0/1376750276 wait complete. 2024-09-16T07:19:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient: build_initial_monmap 2024-09-16T07:19:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient: monmap: 2024-09-16T07:19:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 Processor -- start 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 -- start start 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient: init 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2300a2750 0x7ff2301321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient(hunting): picked mon.noname-b con 0x7ff2300a2750 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2300a3070 0x7ff2301326f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient(hunting): picked mon.noname-c con 0x7ff2300a3070 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff230133730 0x7ff23011bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient(hunting): picked mon.noname-a con 0x7ff230133730 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2301c8fd0 con 0x7ff230133730 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23011f6b0 con 0x7ff2300a2750 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23011fb90 con 0x7ff2300a3070 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22ffff700 10 monclient(hunting): authenticate will time out at 861.419432s 2024-09-16T07:19:00.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff230133730 0x7ff23011bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff2267fc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2300a3070 0x7ff2301326f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff230133730 0x7ff23011bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52682/0 (socket says 172.21.15.37:52682) 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22f7fe700 1 -- 172.21.15.37:0/693696713 learned_addr learned my addr 172.21.15.37:0/693696713 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22f7fe700 10 monclient(hunting): get_auth_request con 0x7ff230133730 auth_method 0 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff2267fc700 10 monclient(hunting): get_auth_request con 0x7ff2300a3070 auth_method 0 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff2267fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff2267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff2267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22effd700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2300a2750 0x7ff2301321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22effd700 10 monclient(hunting): get_auth_request con 0x7ff2300a2750 auth_method 0 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff22effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff2267fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff2267fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.928+0000 7ff2267fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 1 -- 172.21.15.37:0/693696713 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2300a3070 msgr2=0x7ff2301326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2300a3070 0x7ff2301326f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 1 -- 172.21.15.37:0/693696713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2300a2750 msgr2=0x7ff2301321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2300a2750 0x7ff2301321b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 1 monclient: found mon.noname-a 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 1 -- 172.21.15.37:0/693696713 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff228009a70 con 0x7ff230133730 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22f7fe700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff230133730 0x7ff23011bc30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff228014040 tx=0x7ff228004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 1 -- 172.21.15.37:0/693696713 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228012430 con 0x7ff230133730 2024-09-16T07:19:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 10 monclient: dump: 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 10 monclient: _finish_auth 0 2024-09-16T07:19:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 10 monclient: _check_auth_tickets 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 5 monclient: authenticate success, global_id 4356 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 1 -- 172.21.15.37:0/693696713 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff228020070 con 0x7ff230133730 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 10 monclient: _renew_subs 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 1 -- 172.21.15.37:0/693696713 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2280098f0 con 0x7ff230133730 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 1 -- 172.21.15.37:0/693696713 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2301330b0 con 0x7ff230133730 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22dffb700 4 set_mon_vals no callback set 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 10 monclient: dump: 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 10 monclient: _renew_subs 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 1 -- 172.21.15.37:0/693696713 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2300724c0 con 0x7ff230133730 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 1 -- 172.21.15.37:0/693696713 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7ff228004040 con 0x7ff230133730 2024-09-16T07:19:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22ffff700 1 -- 172.21.15.37:0/693696713 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23009cd90 con 0x7ff230133730 2024-09-16T07:19:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff210035630 0x7ff210037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff227fff700 1 -- 172.21.15.37:0/693696713 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4374+0+0 (secure 0 0 0) 0x7ff228022070 con 0x7ff230133730 2024-09-16T07:19:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22effd700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff210035630 0x7ff210037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22effd700 10 monclient: get_auth_request con 0x7ff210035630 auth_method 0 2024-09-16T07:19:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.932+0000 7ff22effd700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff210035630 0x7ff210037af0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff220009bf0 tx=0x7ff220009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.940+0000 7ff227fff700 1 -- 172.21.15.37:0/693696713 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff22801a970 con 0x7ff230133730 2024-09-16T07:19:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.940+0000 7ff227fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:00.940+0000 7ff227fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:01.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.100+0000 7ff22ffff700 1 -- 172.21.15.37:0/693696713 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2300a3480 con 0x7ff210035630 2024-09-16T07:19:01.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.104+0000 7ff227fff700 1 -- 172.21.15.37:0/693696713 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41013 (secure 0 0 0) 0x7ff2200118c0 con 0x7ff210035630 2024-09-16T07:19:01.107 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:19:01.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 -- 172.21.15.37:0/693696713 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff210035630 msgr2=0x7ff210037af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff210035630 0x7ff210037af0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff220009bf0 tx=0x7ff220009240 comp rx=0 tx=0).stop 2024-09-16T07:19:01.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 10 monclient: shutdown 2024-09-16T07:19:01.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:01.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 -- 172.21.15.37:0/693696713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff230133730 msgr2=0x7ff23011bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff230133730 0x7ff23011bc30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff228014040 tx=0x7ff228004af0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 -- 172.21.15.37:0/693696713 shutdown_connections 2024-09-16T07:19:01.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff230133730 0x7ff23011bc30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff210035630 0x7ff210037af0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2300a3070 0x7ff2301326f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 --2- 172.21.15.37:0/693696713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2300a2750 0x7ff2301321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 -- 172.21.15.37:0/693696713 >> 172.21.15.37:0/693696713 conn(0x7ff2300a3ea0 msgr2=0x7ff23006d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:01.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 -- 172.21.15.37:0/693696713 shutdown_connections 2024-09-16T07:19:01.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.108+0000 7ff22ffff700 1 -- 172.21.15.37:0/693696713 wait complete. 2024-09-16T07:19:01.110 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T07:19:01.124 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T07:19:00.588847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164296,"kb_used_data":2152,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066744,"statfs":{"total":579820584960,"available":579652345856,"internally_reserved":0,"allocated":2203648,"data_stored":1432396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":165927590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.136762"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.472033+0000","last_change":"2024-09-16T07:18:57.472278+0000","last_active":"2024-09-16T07:18:57.472033+0000","last_peered":"2024-09-16T07:18:57.472033+0000","last_clean":"2024-09-16T07:18:57.472033+0000","last_became_active":"2024-09-16T07:18:55.463443+0000","last_became_peered":"2024-09-16T07:18:55.463443+0000","last_unstale":"2024-09-16T07:18:57.472033+0000","last_undegraded":"2024-09-16T07:18:57.472033+0000","last_fullsized":"2024-09-16T07:18:57.472033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:10.660959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059939100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.473951+0000","last_change":"2024-09-16T07:18:57.474462+0000","last_active":"2024-09-16T07:18:57.473951+0000","last_peered":"2024-09-16T07:18:57.473951+0000","last_clean":"2024-09-16T07:18:57.473951+0000","last_became_active":"2024-09-16T07:18:55.467864+0000","last_became_peered":"2024-09-16T07:18:55.467864+0000","last_unstale":"2024-09-16T07:18:57.473951+0000","last_undegraded":"2024-09-16T07:18:57.473951+0000","last_fullsized":"2024-09-16T07:18:57.473951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:02.070207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066039799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.471914+0000","last_change":"2024-09-16T07:18:57.472359+0000","last_active":"2024-09-16T07:18:57.471914+0000","last_peered":"2024-09-16T07:18:57.471914+0000","last_clean":"2024-09-16T07:18:57.471914+0000","last_became_active":"2024-09-16T07:18:55.471978+0000","last_became_peered":"2024-09-16T07:18:55.471978+0000","last_unstale":"2024-09-16T07:18:57.471914+0000","last_undegraded":"2024-09-16T07:18:57.471914+0000","last_fullsized":"2024-09-16T07:18:57.471914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:36.770184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077051799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.464442+0000","last_change":"2024-09-16T07:18:57.464442+0000","last_active":"2024-09-16T07:18:57.464442+0000","last_peered":"2024-09-16T07:18:57.464442+0000","last_clean":"2024-09-16T07:18:57.464442+0000","last_became_active":"2024-09-16T07:18:55.468584+0000","last_became_peered":"2024-09-16T07:18:55.468584+0000","last_unstale":"2024-09-16T07:18:57.464442+0000","last_undegraded":"2024-09-16T07:18:57.464442+0000","last_fullsized":"2024-09-16T07:18:57.464442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:20.380838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.470967+0000","last_change":"2024-09-16T07:18:57.470967+0000","last_active":"2024-09-16T07:18:57.470967+0000","last_peered":"2024-09-16T07:18:57.470967+0000","last_clean":"2024-09-16T07:18:57.470967+0000","last_became_active":"2024-09-16T07:18:55.472671+0000","last_became_peered":"2024-09-16T07:18:55.472671+0000","last_unstale":"2024-09-16T07:18:57.470967+0000","last_undegraded":"2024-09-16T07:18:57.470967+0000","last_fullsized":"2024-09-16T07:18:57.470967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:13.933764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000840762,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.739805+0000","last_change":"2024-09-16T07:18:57.740277+0000","last_active":"2024-09-16T07:18:57.739805+0000","last_peered":"2024-09-16T07:18:57.739805+0000","last_clean":"2024-09-16T07:18:57.739805+0000","last_became_active":"2024-09-16T07:18:55.469151+0000","last_became_peered":"2024-09-16T07:18:55.469151+0000","last_unstale":"2024-09-16T07:18:57.739805+0000","last_undegraded":"2024-09-16T07:18:57.739805+0000","last_fullsized":"2024-09-16T07:18:57.739805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:34.482147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064042899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.743056+0000","last_change":"2024-09-16T07:18:57.743569+0000","last_active":"2024-09-16T07:18:57.743056+0000","last_peered":"2024-09-16T07:18:57.743056+0000","last_clean":"2024-09-16T07:18:57.743056+0000","last_became_active":"2024-09-16T07:18:55.469565+0000","last_became_peered":"2024-09-16T07:18:55.469565+0000","last_unstale":"2024-09-16T07:18:57.743056+0000","last_undegraded":"2024-09-16T07:18:57.743056+0000","last_fullsized":"2024-09-16T07:18:57.743056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:46.360051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077454399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.472411+0000","last_change":"2024-09-16T07:18:57.472930+0000","last_active":"2024-09-16T07:18:57.472411+0000","last_peered":"2024-09-16T07:18:57.472411+0000","last_clean":"2024-09-16T07:18:57.472411+0000","last_became_active":"2024-09-16T07:18:55.464362+0000","last_became_peered":"2024-09-16T07:18:55.464362+0000","last_unstale":"2024-09-16T07:18:57.472411+0000","last_undegraded":"2024-09-16T07:18:57.472411+0000","last_fullsized":"2024-09-16T07:18:57.472411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:51.926284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212957,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.401467+0000","last_change":"2024-09-16T07:18:49.375792+0000","last_active":"2024-09-16T07:18:57.401467+0000","last_peered":"2024-09-16T07:18:57.401467+0000","last_clean":"2024-09-16T07:18:57.401467+0000","last_became_active":"2024-09-16T07:18:49.373551+0000","last_became_peered":"2024-09-16T07:18:49.373551+0000","last_unstale":"2024-09-16T07:18:57.401467+0000","last_undegraded":"2024-09-16T07:18:57.401467+0000","last_fullsized":"2024-09-16T07:18:57.401467+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_clean_scrub_stamp":"2024-09-16T07:18:47.386541+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:31.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":62417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":62417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":62436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]}]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":62436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":720896,"data_stored":591345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":720896,"data_stored":591345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:19:01.130 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:19:01.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient: get_monmap_and_config 2024-09-16T07:19:01.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient: build_initial_monmap 2024-09-16T07:19:01.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient: monmap: 2024-09-16T07:19:01.239 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:01.239 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.239 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:01.243649+0000 2024-09-16T07:19:01.240 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:01.243649+0000 2024-09-16T07:19:01.240 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:01.240 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.240 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:01.240 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:01.240 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:01.240 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:01.240 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:01.240 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 Processor -- start 2024-09-16T07:19:01.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 -- start start 2024-09-16T07:19:01.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient: init 2024-09-16T07:19:01.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:01.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T07:19:01.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a0e90 0x7f9a900a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient(hunting): picked mon.noname-b con 0x7f9a900a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:01.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9a900a1270 0x7f9a900a2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient(hunting): picked mon.noname-a con 0x7f9a900a1270 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a900a2b80 0x7f9a900a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient(hunting): picked mon.noname-c con 0x7f9a900a2b80 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9006cf50 con 0x7f9a900a1270 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9009f810 con 0x7f9a900a2b80 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9009f250 con 0x7f9a900a0e90 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient(hunting): authenticate will time out at 861.729463s 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8ffff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9a900a1270 0x7f9a900a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a0e90 0x7f9a900a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8ffff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9a900a1270 0x7f9a900a2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52692/0 (socket says 172.21.15.37:52692) 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a0e90 0x7f9a900a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37914/0 (socket says 172.21.15.37:37914) 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8ffff700 1 -- 172.21.15.37:0/28519360 learned_addr learned my addr 172.21.15.37:0/28519360 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8ffff700 10 monclient(hunting): get_auth_request con 0x7f9a900a1270 auth_method 0 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95451700 1 --2- 172.21.15.37:0/28519360 >> v2:172.21.15.143:3300/0 conn(0x7f9a900a2b80 0x7f9a900a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 10 monclient(hunting): get_auth_request con 0x7f9a900a0e90 auth_method 0 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95451700 10 monclient(hunting): get_auth_request con 0x7f9a900a2b80 auth_method 0 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95451700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95451700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95451700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95451700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95451700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95451700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 1 -- 172.21.15.37:0/28519360 >> v2:172.21.15.143:3300/0 conn(0x7f9a900a2b80 msgr2=0x7f9a900a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 1 --2- 172.21.15.37:0/28519360 >> v2:172.21.15.143:3300/0 conn(0x7f9a900a2b80 0x7f9a900a2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 1 -- 172.21.15.37:0/28519360 >> v2:172.21.15.37:3300/0 conn(0x7f9a900a1270 msgr2=0x7f9a900a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 1 --2- 172.21.15.37:0/28519360 >> v2:172.21.15.37:3300/0 conn(0x7f9a900a1270 0x7f9a900a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 10 monclient: _finish_hunting 0 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 1 monclient: found mon.noname-b 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 1 -- 172.21.15.37:0/28519360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9009f530 con 0x7f9a900a0e90 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a94c50700 1 --2- 172.21.15.37:0/28519360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a0e90 0x7f9a900a04d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9a90067820 tx=0x7f9a78009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee98d7adae8fe787 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 1 -- 172.21.15.37:0/28519360 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78011830 con 0x7f9a900a0e90 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 10 monclient: dump: 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 10 monclient: _finish_auth 0 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:19:01.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 5 monclient: authenticate success, global_id 4319 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 1 -- 172.21.15.37:0/28519360 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a78011990 con 0x7f9a900a0e90 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 4 set_mon_vals no callback set 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 1 -- 172.21.15.37:0/28519360 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78011d20 con 0x7f9a900a0e90 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a8f7fe700 10 monclient: dump: 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.250 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 10 monclient: shutdown 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 -- 172.21.15.37:0/28519360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a0e90 msgr2=0x7f9a900a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 --2- 172.21.15.37:0/28519360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a0e90 0x7f9a900a04d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9a90067820 tx=0x7f9a78009b30 comp rx=0 tx=0).stop 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 -- 172.21.15.37:0/28519360 shutdown_connections 2024-09-16T07:19:01.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 --2- 172.21.15.37:0/28519360 >> v2:172.21.15.143:3300/0 conn(0x7f9a900a2b80 0x7f9a900a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 --2- 172.21.15.37:0/28519360 >> v2:172.21.15.37:3300/0 conn(0x7f9a900a1270 0x7f9a900a2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 --2- 172.21.15.37:0/28519360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a0e90 0x7f9a900a04d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 -- 172.21.15.37:0/28519360 >> 172.21.15.37:0/28519360 conn(0x7f9a900a4740 msgr2=0x7f9a900a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 -- 172.21.15.37:0/28519360 shutdown_connections 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.240+0000 7f9a95c52700 1 -- 172.21.15.37:0/28519360 wait complete. 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient: build_initial_monmap 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient: monmap: 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:01.252 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:01.253 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:01.253 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:01.253 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:01.253 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 1 Processor -- start 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 1 -- start start 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient: init 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a900a0e90 0x7f9a901c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient(hunting): picked mon.noname-a con 0x7f9a900a0e90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a900a1270 0x7f9a901c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient(hunting): picked mon.noname-b con 0x7f9a900a1270 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a2b80 0x7f9a901c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient(hunting): picked mon.noname-c con 0x7f9a900a2b80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9009dfd0 con 0x7f9a900a0e90 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a901c7890 con 0x7f9a900a1270 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9006d780 con 0x7f9a900a2b80 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95c52700 10 monclient(hunting): authenticate will time out at 861.735388s 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a94c50700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a900a0e90 0x7f9a901c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a94c50700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a900a0e90 0x7f9a901c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52708/0 (socket says 172.21.15.37:52708) 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95451700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a2b80 0x7f9a901c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a94c50700 1 -- 172.21.15.37:0/2385123520 learned_addr learned my addr 172.21.15.37:0/2385123520 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a94c50700 10 monclient(hunting): get_auth_request con 0x7f9a900a0e90 auth_method 0 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a94c50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a94c50700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a94c50700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95451700 10 monclient(hunting): get_auth_request con 0x7f9a900a2b80 auth_method 0 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95451700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95451700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95451700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a94c50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a94c50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a8ffff700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a900a1270 0x7f9a901c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a94c50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a8ffff700 10 monclient(hunting): get_auth_request con 0x7f9a900a1270 auth_method 0 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95451700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95451700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a95451700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a8ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a8ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.244+0000 7f9a8ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 1 -- 172.21.15.37:0/2385123520 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a2b80 msgr2=0x7f9a901c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a2b80 0x7f9a901c9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 1 -- 172.21.15.37:0/2385123520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a900a1270 msgr2=0x7f9a901c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a900a1270 0x7f9a901c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 10 monclient: _finish_hunting 0 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 1 monclient: found mon.noname-a 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 1 -- 172.21.15.37:0/2385123520 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780094b0 con 0x7f9a900a0e90 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a94c50700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a900a0e90 0x7f9a901c6920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9a9009f950 tx=0x7f9a78004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 1 -- 172.21.15.37:0/2385123520 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78004450 con 0x7f9a900a0e90 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 10 monclient: dump: 2024-09-16T07:19:01.261 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 10 monclient: _finish_auth 0 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 10 monclient: _check_auth_tickets 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:01.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 1 -- 172.21.15.37:0/2385123520 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7801f070 con 0x7f9a900a0e90 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 5 monclient: authenticate success, global_id 4362 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 10 monclient: _renew_subs 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 1 -- 172.21.15.37:0/2385123520 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a901a8ce0 con 0x7f9a900a0e90 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 1 -- 172.21.15.37:0/2385123520 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78003900 con 0x7f9a900a0e90 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 10 monclient: dump: 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.263 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8effd700 4 set_mon_vals no callback set 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 10 monclient: _renew_subs 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 1 -- 172.21.15.37:0/2385123520 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9006c010 con 0x7f9a900a0e90 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 1 -- 172.21.15.37:0/2385123520 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f9a78003a60 con 0x7f9a900a0e90 2024-09-16T07:19:01.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a95c52700 1 -- 172.21.15.37:0/2385123520 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a900a3610 con 0x7f9a900a0e90 2024-09-16T07:19:01.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9a70035600 0x7f9a70037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8d7fa700 1 -- 172.21.15.37:0/2385123520 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4374+0+0 (secure 0 0 0) 0x7f9a78021070 con 0x7f9a900a0e90 2024-09-16T07:19:01.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8ffff700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9a70035600 0x7f9a70037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8ffff700 10 monclient: get_auth_request con 0x7f9a70035600 auth_method 0 2024-09-16T07:19:01.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.248+0000 7f9a8ffff700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9a70035600 0x7f9a70037ac0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9a90099a40 tx=0x7f9a8000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:01.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.256+0000 7f9a8d7fa700 1 -- 172.21.15.37:0/2385123520 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9a780045d0 con 0x7f9a900a0e90 2024-09-16T07:19:01.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.256+0000 7f9a8d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:01.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.256+0000 7f9a8d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:01.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.424+0000 7f9a95c52700 1 -- 172.21.15.37:0/2385123520 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a900a1670 con 0x7f9a70035600 2024-09-16T07:19:01.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.428+0000 7f9a8d7fa700 1 -- 172.21.15.37:0/2385123520 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41013 (secure 0 0 0) 0x7f9a8000f040 con 0x7f9a70035600 2024-09-16T07:19:01.430 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:19:01.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 -- 172.21.15.37:0/2385123520 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9a70035600 msgr2=0x7f9a70037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9a70035600 0x7f9a70037ac0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9a90099a40 tx=0x7f9a8000b040 comp rx=0 tx=0).stop 2024-09-16T07:19:01.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 10 monclient: shutdown 2024-09-16T07:19:01.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:01.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 -- 172.21.15.37:0/2385123520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a900a0e90 msgr2=0x7f9a901c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a900a0e90 0x7f9a901c6920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9a9009f950 tx=0x7f9a78004770 comp rx=0 tx=0).stop 2024-09-16T07:19:01.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 -- 172.21.15.37:0/2385123520 shutdown_connections 2024-09-16T07:19:01.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f9a70035600 0x7f9a70037ac0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a900a2b80 0x7f9a901c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a900a1270 0x7f9a901c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 --2- 172.21.15.37:0/2385123520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a900a0e90 0x7f9a901c6920 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 -- 172.21.15.37:0/2385123520 >> 172.21.15.37:0/2385123520 conn(0x7f9a900a4740 msgr2=0x7f9a9009c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:01.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 -- 172.21.15.37:0/2385123520 shutdown_connections 2024-09-16T07:19:01.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.432+0000 7f9a95c52700 1 -- 172.21.15.37:0/2385123520 wait complete. 2024-09-16T07:19:01.435 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T07:19:01.447 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T07:19:00.588847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164296,"kb_used_data":2152,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066744,"statfs":{"total":579820584960,"available":579652345856,"internally_reserved":0,"allocated":2203648,"data_stored":1432396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":165927590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.136762"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.472033+0000","last_change":"2024-09-16T07:18:57.472278+0000","last_active":"2024-09-16T07:18:57.472033+0000","last_peered":"2024-09-16T07:18:57.472033+0000","last_clean":"2024-09-16T07:18:57.472033+0000","last_became_active":"2024-09-16T07:18:55.463443+0000","last_became_peered":"2024-09-16T07:18:55.463443+0000","last_unstale":"2024-09-16T07:18:57.472033+0000","last_undegraded":"2024-09-16T07:18:57.472033+0000","last_fullsized":"2024-09-16T07:18:57.472033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:10.660959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059939100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.473951+0000","last_change":"2024-09-16T07:18:57.474462+0000","last_active":"2024-09-16T07:18:57.473951+0000","last_peered":"2024-09-16T07:18:57.473951+0000","last_clean":"2024-09-16T07:18:57.473951+0000","last_became_active":"2024-09-16T07:18:55.467864+0000","last_became_peered":"2024-09-16T07:18:55.467864+0000","last_unstale":"2024-09-16T07:18:57.473951+0000","last_undegraded":"2024-09-16T07:18:57.473951+0000","last_fullsized":"2024-09-16T07:18:57.473951+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:02.070207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066039799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.471914+0000","last_change":"2024-09-16T07:18:57.472359+0000","last_active":"2024-09-16T07:18:57.471914+0000","last_peered":"2024-09-16T07:18:57.471914+0000","last_clean":"2024-09-16T07:18:57.471914+0000","last_became_active":"2024-09-16T07:18:55.471978+0000","last_became_peered":"2024-09-16T07:18:55.471978+0000","last_unstale":"2024-09-16T07:18:57.471914+0000","last_undegraded":"2024-09-16T07:18:57.471914+0000","last_fullsized":"2024-09-16T07:18:57.471914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:36.770184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077051799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.464442+0000","last_change":"2024-09-16T07:18:57.464442+0000","last_active":"2024-09-16T07:18:57.464442+0000","last_peered":"2024-09-16T07:18:57.464442+0000","last_clean":"2024-09-16T07:18:57.464442+0000","last_became_active":"2024-09-16T07:18:55.468584+0000","last_became_peered":"2024-09-16T07:18:55.468584+0000","last_unstale":"2024-09-16T07:18:57.464442+0000","last_undegraded":"2024-09-16T07:18:57.464442+0000","last_fullsized":"2024-09-16T07:18:57.464442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:20.380838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.470967+0000","last_change":"2024-09-16T07:18:57.470967+0000","last_active":"2024-09-16T07:18:57.470967+0000","last_peered":"2024-09-16T07:18:57.470967+0000","last_clean":"2024-09-16T07:18:57.470967+0000","last_became_active":"2024-09-16T07:18:55.472671+0000","last_became_peered":"2024-09-16T07:18:55.472671+0000","last_unstale":"2024-09-16T07:18:57.470967+0000","last_undegraded":"2024-09-16T07:18:57.470967+0000","last_fullsized":"2024-09-16T07:18:57.470967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:13.933764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000840762,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.739805+0000","last_change":"2024-09-16T07:18:57.740277+0000","last_active":"2024-09-16T07:18:57.739805+0000","last_peered":"2024-09-16T07:18:57.739805+0000","last_clean":"2024-09-16T07:18:57.739805+0000","last_became_active":"2024-09-16T07:18:55.469151+0000","last_became_peered":"2024-09-16T07:18:55.469151+0000","last_unstale":"2024-09-16T07:18:57.739805+0000","last_undegraded":"2024-09-16T07:18:57.739805+0000","last_fullsized":"2024-09-16T07:18:57.739805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:34.482147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064042899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.743056+0000","last_change":"2024-09-16T07:18:57.743569+0000","last_active":"2024-09-16T07:18:57.743056+0000","last_peered":"2024-09-16T07:18:57.743056+0000","last_clean":"2024-09-16T07:18:57.743056+0000","last_became_active":"2024-09-16T07:18:55.469565+0000","last_became_peered":"2024-09-16T07:18:55.469565+0000","last_unstale":"2024-09-16T07:18:57.743056+0000","last_undegraded":"2024-09-16T07:18:57.743056+0000","last_fullsized":"2024-09-16T07:18:57.743056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:46.360051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077454399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.472411+0000","last_change":"2024-09-16T07:18:57.472930+0000","last_active":"2024-09-16T07:18:57.472411+0000","last_peered":"2024-09-16T07:18:57.472411+0000","last_clean":"2024-09-16T07:18:57.472411+0000","last_became_active":"2024-09-16T07:18:55.464362+0000","last_became_peered":"2024-09-16T07:18:55.464362+0000","last_unstale":"2024-09-16T07:18:57.472411+0000","last_undegraded":"2024-09-16T07:18:57.472411+0000","last_fullsized":"2024-09-16T07:18:57.472411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:51.926284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212957,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.401467+0000","last_change":"2024-09-16T07:18:49.375792+0000","last_active":"2024-09-16T07:18:57.401467+0000","last_peered":"2024-09-16T07:18:57.401467+0000","last_clean":"2024-09-16T07:18:57.401467+0000","last_became_active":"2024-09-16T07:18:49.373551+0000","last_became_peered":"2024-09-16T07:18:49.373551+0000","last_unstale":"2024-09-16T07:18:57.401467+0000","last_undegraded":"2024-09-16T07:18:57.401467+0000","last_fullsized":"2024-09-16T07:18:57.401467+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_clean_scrub_stamp":"2024-09-16T07:18:47.386541+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:31.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":62417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":62417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":62436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]}]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":62436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":720896,"data_stored":591345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":720896,"data_stored":591345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:19:01.448 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T07:19:01.448 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T07:19:01.448 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T07:19:01.448 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T07:19:01.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient: get_monmap_and_config 2024-09-16T07:19:01.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient: build_initial_monmap 2024-09-16T07:19:01.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient: monmap: 2024-09-16T07:19:01.561 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:01.561 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.561 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:01.564167+0000 2024-09-16T07:19:01.561 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:01.564167+0000 2024-09-16T07:19:01.561 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:01.561 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.561 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:01.561 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:01.561 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:01.561 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:01.562 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:01.562 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 1 Processor -- start 2024-09-16T07:19:01.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 1 -- start start 2024-09-16T07:19:01.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient: init 2024-09-16T07:19:01.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:01.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 0x7f78b01354d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient(hunting): picked mon.noname-b con 0x7f78b01350f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78b0135a10 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f78b012eee0 0x7f78b013d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient(hunting): picked mon.noname-a con 0x7f78b012eee0 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b0074870 con 0x7f78b0135a10 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b0073850 con 0x7f78b012eee0 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b0073c90 con 0x7f78b01350f0 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b5376700 10 monclient(hunting): authenticate will time out at 862.050022s 2024-09-16T07:19:01.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78af7fe700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f78b0135a10 0x7f78b012e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53388/0 (socket says 172.21.15.37:53388) 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78af7fe700 1 -- 172.21.15.37:0/2201055847 learned_addr learned my addr 172.21.15.37:0/2201055847 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b4b75700 1 --2- 172.21.15.37:0/2201055847 >> v2:172.21.15.37:3300/0 conn(0x7f78b012eee0 0x7f78b013d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 1 --2- 172.21.15.37:0/2201055847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 0x7f78b01354d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b4b75700 10 monclient(hunting): get_auth_request con 0x7f78b012eee0 auth_method 0 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b4b75700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b4b75700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b4b75700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 10 monclient(hunting): get_auth_request con 0x7f78b01350f0 auth_method 0 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b4b75700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b4b75700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b4b75700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78aeffd700 1 -- 172.21.15.37:0/2201055847 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987173745 0 0) 0x7f78b0074870 con 0x7f78b0135a10 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78aeffd700 10 monclient(hunting): _negotiate 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78aeffd700 10 monclient(hunting): my global_id is 4368 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78aeffd700 1 -- 172.21.15.37:0/2201055847 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f789c0035d0 con 0x7f78b0135a10 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-09-16T07:19:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 1 -- 172.21.15.37:0/2201055847 >> v2:172.21.15.37:3300/0 conn(0x7f78b012eee0 msgr2=0x7f78b013d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78aeffd700 1 -- 172.21.15.37:0/2201055847 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3028972918 0 0) 0x7f789c0035d0 con 0x7f78b0135a10 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 1 --2- 172.21.15.37:0/2201055847 >> v2:172.21.15.37:3300/0 conn(0x7f78b012eee0 0x7f78b013d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 1 -- 172.21.15.37:0/2201055847 >> v1:172.21.15.37:6789/0 conn(0x7f78b0135a10 legacy=0x7f78b012e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 10 monclient: _finish_hunting 0 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 1 monclient: found mon.noname-b 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 1 -- 172.21.15.37:0/2201055847 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b0099bf0 con 0x7f78b01350f0 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78aeffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78b4b75700 1 --2- 172.21.15.37:0/2201055847 >> v2:172.21.15.37:3300/0 conn(0x7f78b012eee0 0x7f78b013d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.560+0000 7f78affff700 1 --2- 172.21.15.37:0/2201055847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 0x7f78b01354d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7898009850 tx=0x7f7898009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3af0aa6630ebb12 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 1 -- 172.21.15.37:0/2201055847 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0001670 con 0x7f78b01350f0 2024-09-16T07:19:01.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 10 monclient: dump: 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 10 monclient: _finish_auth 0 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 5 monclient: authenticate success, global_id 4325 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 1 -- 172.21.15.37:0/2201055847 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7898004030 con 0x7f78b01350f0 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 4 set_mon_vals no callback set 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 1 -- 172.21.15.37:0/2201055847 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980043a0 con 0x7f78b01350f0 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78aeffd700 10 monclient: dump: 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.581 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient: shutdown 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 -- 172.21.15.37:0/2201055847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 msgr2=0x7f78b01354d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 --2- 172.21.15.37:0/2201055847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 0x7f78b01354d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7898009850 tx=0x7f7898009b60 comp rx=0 tx=0).stop 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 -- 172.21.15.37:0/2201055847 shutdown_connections 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 --2- 172.21.15.37:0/2201055847 >> v2:172.21.15.37:3300/0 conn(0x7f78b012eee0 0x7f78b013d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 --2- 172.21.15.37:0/2201055847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 0x7f78b01354d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 -- 172.21.15.37:0/2201055847 >> 172.21.15.37:0/2201055847 conn(0x7f78b00a0b40 msgr2=0x7f78b00a0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 -- 172.21.15.37:0/2201055847 shutdown_connections 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 -- 172.21.15.37:0/2201055847 wait complete. 2024-09-16T07:19:01.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient: build_initial_monmap 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient: monmap: 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:01.583 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 Processor -- start 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 -- start start 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient: init 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78b012eee0 0x7f78b011b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient(hunting): picked mon.noname-a con 0x7f78b012eee0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 0x7f78b011b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient(hunting): picked mon.noname-c con 0x7f78b01350f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b0135a10 0x7f78b011be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient(hunting): picked mon.noname-b con 0x7f78b0135a10 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:01.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b012ded0 con 0x7f78b012eee0 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b009b230 con 0x7f78b0135a10 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b011f130 con 0x7f78b01350f0 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78b5376700 10 monclient(hunting): authenticate will time out at 862.055880s 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.564+0000 7f78affff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78b012eee0 0x7f78b011b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78b012eee0 0x7f78b011b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52732/0 (socket says 172.21.15.37:52732) 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 1 -- 172.21.15.37:0/375553656 learned_addr learned my addr 172.21.15.37:0/375553656 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:01.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 10 monclient(hunting): get_auth_request con 0x7f78b012eee0 auth_method 0 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78af7fe700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 0x7f78b011b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78af7fe700 10 monclient(hunting): get_auth_request con 0x7f78b01350f0 auth_method 0 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b4b75700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b0135a10 0x7f78b011be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2024-09-16T07:19:01.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 1 -- 172.21.15.37:0/375553656 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 msgr2=0x7f78b011b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 0x7f78b011b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 1 -- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b0135a10 msgr2=0x7f78b011be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b0135a10 0x7f78b011be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 10 monclient: _finish_hunting 0 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 1 monclient: found mon.noname-a 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 1 -- 172.21.15.37:0/375553656 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a8009a70 con 0x7f78b012eee0 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b4b75700 10 monclient: get_auth_request con 0x7f78b0135a10 auth_method 0 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b4b75700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b0135a10 0x7f78b011be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78af7fe700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 0x7f78b011b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:19:01.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78affff700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78b012eee0 0x7f78b011b3a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f78980090b0 tx=0x7f7898004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 1 -- 172.21.15.37:0/375553656 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898004590 con 0x7f78b012eee0 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 10 monclient: dump: 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.588 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 10 monclient: _finish_auth 0 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 10 monclient: _check_auth_tickets 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 5 monclient: authenticate success, global_id 4371 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 10 monclient: _renew_subs 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 1 -- 172.21.15.37:0/375553656 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78980094b0 con 0x7f78b012eee0 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 1 -- 172.21.15.37:0/375553656 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7898011630 con 0x7f78b012eee0 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 1 -- 172.21.15.37:0/375553656 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980119c0 con 0x7f78b012eee0 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 10 monclient: dump: 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 10 monclient: _renew_subs 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:01.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 1 -- 172.21.15.37:0/375553656 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b009fbe0 con 0x7f78b012eee0 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78ae7fc700 4 set_mon_vals no callback set 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78b5376700 1 -- 172.21.15.37:0/375553656 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b00727c0 con 0x7f78b012eee0 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 1 -- 172.21.15.37:0/375553656 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f7898021070 con 0x7f78b012eee0 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7890035600 0x7f7890037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78acff9700 1 -- 172.21.15.37:0/375553656 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4374+0+0 (secure 0 0 0) 0x7f7898049110 con 0x7f78b012eee0 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78af7fe700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7890035600 0x7f7890037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.568+0000 7f78af7fe700 10 monclient: get_auth_request con 0x7f7890035600 auth_method 0 2024-09-16T07:19:01.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.572+0000 7f78af7fe700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7890035600 0x7f7890037ac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78a0001860 tx=0x7f78a00015d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:01.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.576+0000 7f78acff9700 1 -- 172.21.15.37:0/375553656 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f789802d3a0 con 0x7f78b012eee0 2024-09-16T07:19:01.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.576+0000 7f78acff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:01.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.576+0000 7f78acff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:01.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.776+0000 7f78b5376700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T07:19:01.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.776+0000 7f78b5376700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T07:19:01.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.776+0000 7f78b5376700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:01.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.776+0000 7f78b5376700 1 -- 172.21.15.37:0/375553656 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f78b009ab90 con 0x7f78b012eee0 2024-09-16T07:19:01.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.776+0000 7f78acff9700 1 -- 172.21.15.37:0/375553656 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7898029080 con 0x7f78b012eee0 2024-09-16T07:19:01.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.776+0000 7f78acff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T07:19:01.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.776+0000 7f78acff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:01.777 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:19:01.777 INFO:teuthology.orchestra.run.smithi037.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T07:19:01.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 -- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7890035600 msgr2=0x7f7890037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7890035600 0x7f7890037ac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f78a0001860 tx=0x7f78a00015d0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 10 monclient: shutdown 2024-09-16T07:19:01.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:01.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 -- 172.21.15.37:0/375553656 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78b012eee0 msgr2=0x7f78b011b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78b012eee0 0x7f78b011b3a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f78980090b0 tx=0x7f7898004770 comp rx=0 tx=0).stop 2024-09-16T07:19:01.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 -- 172.21.15.37:0/375553656 shutdown_connections 2024-09-16T07:19:01.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b0135a10 0x7f78b011be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7890035600 0x7f7890037ac0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78b01350f0 0x7f78b011b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 --2- 172.21.15.37:0/375553656 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78b012eee0 0x7f78b011b3a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 -- 172.21.15.37:0/375553656 >> 172.21.15.37:0/375553656 conn(0x7f78b00a0b40 msgr2=0x7f78b01345c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:01.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 -- 172.21.15.37:0/375553656 shutdown_connections 2024-09-16T07:19:01.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.780+0000 7f78b5376700 1 -- 172.21.15.37:0/375553656 wait complete. 2024-09-16T07:19:01.790 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T07:19:01.790 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T07:19:01.805 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T07:19:01.805 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T07:19:01.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.920+0000 7f47ba344700 10 monclient: get_monmap_and_config 2024-09-16T07:19:01.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.920+0000 7f47ba344700 10 monclient: build_initial_monmap 2024-09-16T07:19:01.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.920+0000 7f47ba344700 10 monclient: monmap: 2024-09-16T07:19:01.922 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:01.922 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.922 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:01.926390+0000 2024-09-16T07:19:01.922 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:01.926390+0000 2024-09-16T07:19:01.922 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:01.922 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.922 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:01.923 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:01.923 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:01.923 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:01.923 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:01.923 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.920+0000 7f47ba344700 1 Processor -- start 2024-09-16T07:19:01.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.920+0000 7f47ba344700 1 -- start start 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.920+0000 7f47ba344700 10 monclient: init 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f47ac0b63f0 0x7f47ac0234f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient(hunting): picked mon.noname-a con 0x7f47ac0b63f0 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47ac0b67d0 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f47ac0b6c20 0x7f47ac0b9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient(hunting): picked mon.noname-c con 0x7f47ac0b6c20 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac0b6030 con 0x7f47ac0b67d0 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ac020000 con 0x7f47ac0b63f0 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ac01fce0 con 0x7f47ac0b6c20 2024-09-16T07:19:01.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient(hunting): authenticate will time out at 862.412236s 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f47ac0b63f0 0x7f47ac0234f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f47ac0b63f0 0x7f47ac0234f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52746/0 (socket says 172.21.15.37:52746) 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b8b41700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f47ac0b67d0 0x7f47ac0b9530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53400/0 (socket says 172.21.15.37:53400) 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 1 -- 172.21.15.37:0/330857907 learned_addr learned my addr 172.21.15.37:0/330857907 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 10 monclient(hunting): get_auth_request con 0x7f47ac0b63f0 auth_method 0 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9b43700 1 --2- 172.21.15.37:0/330857907 >> v2:172.21.15.143:3300/0 conn(0x7f47ac0b6c20 0x7f47ac0b9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9b43700 10 monclient(hunting): get_auth_request con 0x7f47ac0b6c20 auth_method 0 2024-09-16T07:19:01.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9b43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9b43700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9b43700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9b43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9b43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9b43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 1 -- 172.21.15.37:0/330857907 >> v2:172.21.15.143:3300/0 conn(0x7f47ac0b6c20 msgr2=0x7f47ac0b9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 1 --2- 172.21.15.37:0/330857907 >> v2:172.21.15.143:3300/0 conn(0x7f47ac0b6c20 0x7f47ac0b9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 1 -- 172.21.15.37:0/330857907 >> v1:172.21.15.37:6789/0 conn(0x7f47ac0b67d0 legacy=0x7f47ac0b9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 10 monclient: _finish_hunting 0 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 1 monclient: found mon.noname-a 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 1 -- 172.21.15.37:0/330857907 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac0222e0 con 0x7f47ac0b63f0 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47b9342700 1 --2- 172.21.15.37:0/330857907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b63f0 0x7f47ac0234f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0b7090 tx=0x7f47b0009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c734f050ac4a0598 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 1 -- 172.21.15.37:0/330857907 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0011830 con 0x7f47ac0b63f0 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 10 monclient: dump: 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 10 monclient: _finish_auth 0 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 10 monclient: _check_auth_tickets 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 5 monclient: authenticate success, global_id 4374 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 1 -- 172.21.15.37:0/330857907 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b0011990 con 0x7f47ac0b63f0 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 1 -- 172.21.15.37:0/330857907 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b0011d20 con 0x7f47ac0b63f0 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47abfff700 10 monclient: dump: 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.933 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 4 set_mon_vals no callback set 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 10 monclient: shutdown 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 -- 172.21.15.37:0/330857907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b63f0 msgr2=0x7f47ac0234f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 --2- 172.21.15.37:0/330857907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b63f0 0x7f47ac0234f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0b7090 tx=0x7f47b0009bf0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 -- 172.21.15.37:0/330857907 shutdown_connections 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 --2- 172.21.15.37:0/330857907 >> v2:172.21.15.143:3300/0 conn(0x7f47ac0b6c20 0x7f47ac0b9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 --2- 172.21.15.37:0/330857907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b63f0 0x7f47ac0234f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 -- 172.21.15.37:0/330857907 >> 172.21.15.37:0/330857907 conn(0x7f47ac0064b0 msgr2=0x7f47ac0068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 -- 172.21.15.37:0/330857907 shutdown_connections 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.924+0000 7f47ba344700 1 -- 172.21.15.37:0/330857907 wait complete. 2024-09-16T07:19:01.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient: build_initial_monmap 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient: monmap: 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:01.935 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 1 Processor -- start 2024-09-16T07:19:01.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 1 -- start start 2024-09-16T07:19:01.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient: init 2024-09-16T07:19:01.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:01.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47ac0b63f0 0x7f47ac0baca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient(hunting): picked mon.noname-b con 0x7f47ac0b63f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f47ac0b67d0 0x7f47ac0bb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient(hunting): picked mon.noname-c con 0x7f47ac0b67d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b6c20 0x7f47ac0bb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient(hunting): picked mon.noname-a con 0x7f47ac0b6c20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac0c06d0 con 0x7f47ac0b6c20 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac0be0f0 con 0x7f47ac0b63f0 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac0bd420 con 0x7f47ac0b67d0 2024-09-16T07:19:01.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47ba344700 10 monclient(hunting): authenticate will time out at 862.419207s 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f47ac0b67d0 0x7f47ac0bb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9b43700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b6c20 0x7f47ac0bb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f47ac0b67d0 0x7f47ac0bb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37946/0 (socket says 172.21.15.37:37946) 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9342700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47ac0b63f0 0x7f47ac0baca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9b43700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b6c20 0x7f47ac0bb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52760/0 (socket says 172.21.15.37:52760) 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 1 -- 172.21.15.37:0/1281274980 learned_addr learned my addr 172.21.15.37:0/1281274980 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 10 monclient(hunting): get_auth_request con 0x7f47ac0b67d0 auth_method 0 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9b43700 10 monclient(hunting): get_auth_request con 0x7f47ac0b6c20 auth_method 0 2024-09-16T07:19:01.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9b43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9b43700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9b43700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9342700 10 monclient(hunting): get_auth_request con 0x7f47ac0b63f0 auth_method 0 2024-09-16T07:19:01.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9342700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:01.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9342700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:01.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9342700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:01.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9b43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9b43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9b43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9342700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9342700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b9342700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 1 -- 172.21.15.37:0/1281274980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47ac0b63f0 msgr2=0x7f47ac0baca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47ac0b63f0 0x7f47ac0baca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 1 -- 172.21.15.37:0/1281274980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b6c20 msgr2=0x7f47ac0bb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b6c20 0x7f47ac0bb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 10 monclient: _finish_hunting 0 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 1 monclient: found mon.noname-c 2024-09-16T07:19:01.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.928+0000 7f47b8b41700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:01.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47b8b41700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:01.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47b8b41700 1 -- 172.21.15.37:0/1281274980 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b0009570 con 0x7f47ac0b67d0 2024-09-16T07:19:01.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47b9b43700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b6c20 0x7f47ac0bb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:19:01.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47b8b41700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f47ac0b67d0 0x7f47ac0bb1e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f47a400c530 tx=0x7f47a400c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:01.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 1 -- 172.21.15.37:0/1281274980 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a4005cd0 con 0x7f47ac0b67d0 2024-09-16T07:19:01.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:01.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 10 monclient: dump: 2024-09-16T07:19:01.946 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 10 monclient: _finish_auth 0 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 1 -- 172.21.15.37:0/1281274980 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a4005e30 con 0x7f47ac0b67d0 2024-09-16T07:19:01.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 5 monclient: authenticate success, global_id 4331 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 10 monclient: _renew_subs 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 1 -- 172.21.15.37:0/1281274980 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ac028ce0 con 0x7f47ac0b67d0 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 1 -- 172.21.15.37:0/1281274980 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a40072f0 con 0x7f47ac0b67d0 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 10 monclient: dump: 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:01.948 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:01.949 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:01.949 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:01.949 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:01.949 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:01.949 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:01.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ab7fe700 4 set_mon_vals no callback set 2024-09-16T07:19:01.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 10 monclient: _renew_subs 2024-09-16T07:19:01.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:01.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 1 -- 172.21.15.37:0/1281274980 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ac020870 con 0x7f47ac0b67d0 2024-09-16T07:19:01.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:01.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:01.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:01.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47ba344700 1 -- 172.21.15.37:0/1281274980 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ac0be310 con 0x7f47ac0b67d0 2024-09-16T07:19:01.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 1 -- 172.21.15.37:0/1281274980 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f47a4007450 con 0x7f47ac0b67d0 2024-09-16T07:19:01.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4798035600 0x7f4798037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:01.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47a9ffb700 1 -- 172.21.15.37:0/1281274980 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4374+0+0 (secure 0 0 0) 0x7f47a4046dc0 con 0x7f47ac0b67d0 2024-09-16T07:19:01.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47b9342700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4798035600 0x7f4798037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:01.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47b9342700 10 monclient: get_auth_request con 0x7f4798035600 auth_method 0 2024-09-16T07:19:01.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.932+0000 7f47b9342700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4798035600 0x7f4798037ac0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f47b0000c00 tx=0x7f47b0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:01.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.940+0000 7f47a9ffb700 1 -- 172.21.15.37:0/1281274980 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f47ac0be310 con 0x7f47ac0b67d0 2024-09-16T07:19:01.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.940+0000 7f47a9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:01.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:01.940+0000 7f47a9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:02.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.112+0000 7f47ba344700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:19:02.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.112+0000 7f47ba344700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:19:02.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.112+0000 7f47ba344700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:02.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.112+0000 7f47ba344700 1 -- 172.21.15.37:0/1281274980 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47ac0be310 con 0x7f47ac0b67d0 2024-09-16T07:19:02.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.112+0000 7f47a9ffb700 1 -- 172.21.15.37:0/1281274980 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+12233 (secure 0 0 0) 0x7f47ac0be310 con 0x7f47ac0b67d0 2024-09-16T07:19:02.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.112+0000 7f47a9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:19:02.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.112+0000 7f47a9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:02.114 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:19:02.114 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":21,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","created":"2024-09-16T07:18:36.646941+0000","modified":"2024-09-16T07:19:00.588595+0000","last_up_change":"2024-09-16T07:18:46.372944+0000","last_in_change":"2024-09-16T07:18:38.297482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T07:18:46.507362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T07:18:53.543683+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd764fc8-b5fb-40fe-a64f-ce7b8c1df525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6801","nonce":3137250735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6803","nonce":3137250735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6807","nonce":3137250735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6805","nonce":3137250735}]},"public_addr":"172.21.15.37:6801/3137250735","cluster_addr":"172.21.15.37:6803/3137250735","heartbeat_back_addr":"172.21.15.37:6807/3137250735","heartbeat_front_addr":"172.21.15.37:6805/3137250735","state":["exists","up"]},{"osd":1,"uuid":"52d577a5-08b7-452d-aa35-e1560215a09e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6809","nonce":810396637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6811","nonce":810396637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6815","nonce":810396637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6813","nonce":810396637}]},"public_addr":"172.21.15.37:6809/810396637","cluster_addr":"172.21.15.37:6811/810396637","heartbeat_back_addr":"172.21.15.37:6815/810396637","heartbeat_front_addr":"172.21.15.37:6813/810396637","state":["exists","up"]},{"osd":2,"uuid":"e3deb985-fb4e-42ee-8ed6-e6318cbca322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6817","nonce":265940208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6819","nonce":265940208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6823","nonce":265940208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6821","nonce":265940208}]},"public_addr":"172.21.15.37:6817/265940208","cluster_addr":"172.21.15.37:6819/265940208","heartbeat_back_addr":"172.21.15.37:6823/265940208","heartbeat_front_addr":"172.21.15.37:6821/265940208","state":["exists","up"]},{"osd":3,"uuid":"ab447540-8ed1-4a0c-8f6c-dfeea8af02ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6811","nonce":2588720019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6815","nonce":2588720019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6823","nonce":2588720019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6819","nonce":2588720019}]},"public_addr":"172.21.15.143:6811/2588720019","cluster_addr":"172.21.15.143:6815/2588720019","heartbeat_back_addr":"172.21.15.143:6823/2588720019","heartbeat_front_addr":"172.21.15.143:6819/2588720019","state":["exists","up"]},{"osd":4,"uuid":"be421fc0-1edf-4f20-b530-c9cf4eb20a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6810","nonce":3355421705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6814","nonce":3355421705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6822","nonce":3355421705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6818","nonce":3355421705}]},"public_addr":"172.21.15.143:6810/3355421705","cluster_addr":"172.21.15.143:6814/3355421705","heartbeat_back_addr":"172.21.15.143:6822/3355421705","heartbeat_front_addr":"172.21.15.143:6818/3355421705","state":["exists","up"]},{"osd":5,"uuid":"8c7725bd-5f56-4fa0-b455-6569e313f9fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6801","nonce":3921632375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6803","nonce":3921632375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6807","nonce":3921632375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6805","nonce":3921632375}]},"public_addr":"172.21.15.143:6801/3921632375","cluster_addr":"172.21.15.143:6803/3921632375","heartbeat_back_addr":"172.21.15.143:6807/3921632375","heartbeat_front_addr":"172.21.15.143:6805/3921632375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.492199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.565060+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.519201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.777218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.842229+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.307398+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:19:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 -- 172.21.15.37:0/1281274980 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4798035600 msgr2=0x7f4798037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4798035600 0x7f4798037ac0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f47b0000c00 tx=0x7f47b0008030 comp rx=0 tx=0).stop 2024-09-16T07:19:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 10 monclient: shutdown 2024-09-16T07:19:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:02.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 -- 172.21.15.37:0/1281274980 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f47ac0b67d0 msgr2=0x7f47ac0bb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f47ac0b67d0 0x7f47ac0bb1e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f47a400c530 tx=0x7f47a400c840 comp rx=0 tx=0).stop 2024-09-16T07:19:02.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 -- 172.21.15.37:0/1281274980 shutdown_connections 2024-09-16T07:19:02.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f47ac0b6c20 0x7f47ac0bb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4798035600 0x7f4798037ac0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f47ac0b67d0 0x7f47ac0bb1e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 --2- 172.21.15.37:0/1281274980 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47ac0b63f0 0x7f47ac0baca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 -- 172.21.15.37:0/1281274980 >> 172.21.15.37:0/1281274980 conn(0x7f47ac0064b0 msgr2=0x7f47ac01d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:02.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 -- 172.21.15.37:0/1281274980 shutdown_connections 2024-09-16T07:19:02.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.116+0000 7f47ba344700 1 -- 172.21.15.37:0/1281274980 wait complete. 2024-09-16T07:19:02.128 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T07:18:46.507362+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T07:18:53.543683+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T07:19:02.128 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T07:19:02.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.240+0000 7f229ce50700 10 monclient: get_monmap_and_config 2024-09-16T07:19:02.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.240+0000 7f229ce50700 10 monclient: build_initial_monmap 2024-09-16T07:19:02.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.240+0000 7f229ce50700 10 monclient: monmap: 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:02.246884+0000 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:02.246884+0000 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:02.243 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 Processor -- start 2024-09-16T07:19:02.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 -- start start 2024-09-16T07:19:02.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient: init 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f22980a2750 0x7f22980a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient(hunting): picked mon.noname-a con 0x7f22980a2750 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22980a3070 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2298133730 0x7f229813b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient(hunting): picked mon.noname-c con 0x7f2298133730 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2298099be0 con 0x7f22980a3070 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229809de40 con 0x7f22980a2750 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229809d530 con 0x7f2298133730 2024-09-16T07:19:02.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient(hunting): authenticate will time out at 862.732998s 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f22980a2750 0x7f22980a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f22980a2750 0x7f22980a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52776/0 (socket says 172.21.15.37:52776) 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 1 -- 172.21.15.37:0/799637999 learned_addr learned my addr 172.21.15.37:0/799637999 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 10 monclient(hunting): get_auth_request con 0x7f22980a2750 auth_method 0 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2297fff700 1 --2- 172.21.15.37:0/799637999 >> v2:172.21.15.143:3300/0 conn(0x7f2298133730 0x7f229813b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 10 monclient(hunting): handle_auth_done global_id 4383 payload 274 2024-09-16T07:19:02.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 1 -- 172.21.15.37:0/799637999 >> v2:172.21.15.143:3300/0 conn(0x7f2298133730 msgr2=0x7f229813b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:02.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 1 --2- 172.21.15.37:0/799637999 >> v2:172.21.15.143:3300/0 conn(0x7f2298133730 0x7f229813b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 1 -- 172.21.15.37:0/799637999 >> v1:172.21.15.143:6789/0 conn(0x7f22980a3070 legacy=0x7f2298133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 10 monclient: _finish_hunting 0 2024-09-16T07:19:02.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 1 monclient: found mon.noname-a 2024-09-16T07:19:02.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:02.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 1 -- 172.21.15.37:0/799637999 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229809a020 con 0x7f22980a2750 2024-09-16T07:19:02.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2297fff700 10 monclient: get_auth_request con 0x7f2298133730 auth_method 0 2024-09-16T07:19:02.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2297fff700 1 --2- 172.21.15.37:0/799637999 >> v2:172.21.15.143:3300/0 conn(0x7f2298133730 0x7f229813b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T07:19:02.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f22977fe700 1 --2- 172.21.15.37:0/799637999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a2750 0x7f22980a2b30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f22980711d0 tx=0x7f2288009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8d6ba9eafa7036 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:02.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 1 -- 172.21.15.37:0/799637999 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228800b8a0 con 0x7f22980a2750 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 10 monclient: dump: 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.251 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 10 monclient: _finish_auth 0 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 1 -- 172.21.15.37:0/799637999 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228800ba00 con 0x7f22980a2750 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 5 monclient: authenticate success, global_id 4383 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 1 -- 172.21.15.37:0/799637999 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228800bd90 con 0x7f22980a2750 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 4 set_mon_vals no callback set 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:02.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f2296ffd700 10 monclient: dump: 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 10 monclient: shutdown 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:02.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 -- 172.21.15.37:0/799637999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a2750 msgr2=0x7f22980a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 --2- 172.21.15.37:0/799637999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a2750 0x7f22980a2b30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f22980711d0 tx=0x7f2288009bf0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 -- 172.21.15.37:0/799637999 shutdown_connections 2024-09-16T07:19:02.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 --2- 172.21.15.37:0/799637999 >> v2:172.21.15.143:3300/0 conn(0x7f2298133730 0x7f229813b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 --2- 172.21.15.37:0/799637999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a2750 0x7f22980a2b30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 -- 172.21.15.37:0/799637999 >> 172.21.15.37:0/799637999 conn(0x7f22980a3ea0 msgr2=0x7f22980a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:02.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 -- 172.21.15.37:0/799637999 shutdown_connections 2024-09-16T07:19:02.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.244+0000 7f229ce50700 1 -- 172.21.15.37:0/799637999 wait complete. 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient: build_initial_monmap 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient: monmap: 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:02.260 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 1 Processor -- start 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 1 -- start start 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient: init 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f22980a2750 0x7f2298132580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient(hunting): picked mon.noname-c con 0x7f22980a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a3070 0x7f2298132ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient(hunting): picked mon.noname-a con 0x7f22980a3070 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2298133730 0x7f229809ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient(hunting): picked mon.noname-b con 0x7f2298133730 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:02.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22980724c0 con 0x7f22980a3070 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22981cb050 con 0x7f2298133730 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f229806d1d0 con 0x7f22980a2750 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient(hunting): authenticate will time out at 862.737778s 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a3070 0x7f2298132ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a3070 0x7f2298132ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52784/0 (socket says 172.21.15.37:52784) 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 1 -- 172.21.15.37:0/3450943286 learned_addr learned my addr 172.21.15.37:0/3450943286 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 10 monclient(hunting): get_auth_request con 0x7f22980a3070 auth_method 0 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f22977fe700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f22980a2750 0x7f2298132580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2297fff700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2298133730 0x7f229809ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f22977fe700 10 monclient(hunting): get_auth_request con 0x7f22980a2750 auth_method 0 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f22977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f22977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f22977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2297fff700 10 monclient(hunting): get_auth_request con 0x7f2298133730 auth_method 0 2024-09-16T07:19:02.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2297fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2297fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2297fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f22977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f22977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f22977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2297fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2297fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2297fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 1 -- 172.21.15.37:0/3450943286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f22980a2750 msgr2=0x7f2298132580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f22980a2750 0x7f2298132580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 1 -- 172.21.15.37:0/3450943286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2298133730 msgr2=0x7f229809ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2298133730 0x7f229809ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 10 monclient: _finish_hunting 0 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 1 monclient: found mon.noname-a 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 1 -- 172.21.15.37:0/3450943286 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2288009570 con 0x7f22980a3070 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f228effd700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a3070 0x7f2298132ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f228000c500 tx=0x7f228000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 1 -- 172.21.15.37:0/3450943286 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22800052d0 con 0x7f22980a3070 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 10 monclient: dump: 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.265 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 10 monclient: _finish_auth 0 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 5 monclient: authenticate success, global_id 4386 2024-09-16T07:19:02.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient: _renew_subs 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 1 -- 172.21.15.37:0/3450943286 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2280005430 con 0x7f22980a3070 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 1 -- 172.21.15.37:0/3450943286 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22981330b0 con 0x7f22980a3070 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 1 -- 172.21.15.37:0/3450943286 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22800055a0 con 0x7f22980a3070 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient: _renew_subs 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f229ce50700 1 -- 172.21.15.37:0/3450943286 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229809c680 con 0x7f22980a3070 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f22967fc700 4 set_mon_vals no callback set 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.248+0000 7f2294ff9700 10 monclient: dump: 2024-09-16T07:19:02.267 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.252+0000 7f2294ff9700 1 -- 172.21.15.37:0/3450943286 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f22800036a0 con 0x7f22980a3070 2024-09-16T07:19:02.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.252+0000 7f229ce50700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:02.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.252+0000 7f229ce50700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:02.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.252+0000 7f229ce50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.252+0000 7f229ce50700 1 -- 172.21.15.37:0/3450943286 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229809cd90 con 0x7f22980a3070 2024-09-16T07:19:02.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.252+0000 7f2294ff9700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2278035600 0x7f2278037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.252+0000 7f2294ff9700 1 -- 172.21.15.37:0/3450943286 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4374+0+0 (secure 0 0 0) 0x7f2280046d30 con 0x7f22980a3070 2024-09-16T07:19:02.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.252+0000 7f22977fe700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2278035600 0x7f2278037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.252+0000 7f22977fe700 10 monclient: get_auth_request con 0x7f2278035600 auth_method 0 2024-09-16T07:19:02.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.252+0000 7f22977fe700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2278035600 0x7f2278037ac0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2288000c00 tx=0x7f2288008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:02.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.260+0000 7f2294ff9700 1 -- 172.21.15.37:0/3450943286 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f22800108e0 con 0x7f22980a3070 2024-09-16T07:19:02.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.260+0000 7f2294ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:02.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.260+0000 7f2294ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.424+0000 7f229ce50700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T07:19:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.424+0000 7f229ce50700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T07:19:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.424+0000 7f229ce50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.424+0000 7f229ce50700 1 -- 172.21.15.37:0/3450943286 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2298070d70 con 0x7f22980a3070 2024-09-16T07:19:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.424+0000 7f2294ff9700 1 -- 172.21.15.37:0/3450943286 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2280010700 con 0x7f22980a3070 2024-09-16T07:19:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.424+0000 7f2294ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T07:19:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.424+0000 7f2294ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:02.426 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-09-16T07:19:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 -- 172.21.15.37:0/3450943286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2278035600 msgr2=0x7f2278037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2278035600 0x7f2278037ac0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2288000c00 tx=0x7f2288008030 comp rx=0 tx=0).stop 2024-09-16T07:19:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 10 monclient: shutdown 2024-09-16T07:19:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 -- 172.21.15.37:0/3450943286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a3070 msgr2=0x7f2298132ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a3070 0x7f2298132ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f228000c500 tx=0x7f228000c810 comp rx=0 tx=0).stop 2024-09-16T07:19:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 -- 172.21.15.37:0/3450943286 shutdown_connections 2024-09-16T07:19:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2298133730 0x7f229809ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22980a3070 0x7f2298132ac0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2278035600 0x7f2278037ac0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 --2- 172.21.15.37:0/3450943286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f22980a2750 0x7f2298132580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 -- 172.21.15.37:0/3450943286 >> 172.21.15.37:0/3450943286 conn(0x7f22980a3ea0 msgr2=0x7f229809f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 -- 172.21.15.37:0/3450943286 shutdown_connections 2024-09-16T07:19:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.428+0000 7f229ce50700 1 -- 172.21.15.37:0/3450943286 wait complete. 2024-09-16T07:19:02.441 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T07:19:02.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.556+0000 7fbb3f54d700 10 monclient: get_monmap_and_config 2024-09-16T07:19:02.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.556+0000 7fbb3f54d700 10 monclient: build_initial_monmap 2024-09-16T07:19:02.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.556+0000 7fbb3f54d700 10 monclient: monmap: 2024-09-16T07:19:02.558 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:02.558 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.558 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:02.562546+0000 2024-09-16T07:19:02.559 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:02.562546+0000 2024-09-16T07:19:02.559 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:02.559 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.559 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:02.559 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:02.559 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:02.559 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:02.559 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:02.559 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.556+0000 7fbb3f54d700 1 Processor -- start 2024-09-16T07:19:02.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.556+0000 7fbb3f54d700 1 -- start start 2024-09-16T07:19:02.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.556+0000 7fbb3f54d700 10 monclient: init 2024-09-16T07:19:02.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:02.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T07:19:02.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb380a2750 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:02.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb38071880 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:02.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbb380a31e0 0x7fbb3813a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient(hunting): picked mon.noname-a con 0x7fbb380a31e0 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb380a2b30 con 0x7fbb38071880 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3809c3e0 con 0x7fbb380a2750 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3809d0e0 con 0x7fbb380a31e0 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient(hunting): authenticate will time out at 863.048761s 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbb380a31e0 0x7fbb3813a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3dd4a700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fbb38071880 0x7fbb3809a0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53416/0 (socket says 172.21.15.37:53416) 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3dd4a700 1 -- 172.21.15.37:0/705681143 learned_addr learned my addr 172.21.15.37:0/705681143 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 10 monclient(hunting): get_auth_request con 0x7fbb380a31e0 auth_method 0 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 10 monclient(hunting): handle_auth_done global_id 4389 payload 274 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 1 -- 172.21.15.37:0/705681143 >> v1:172.21.15.143:6789/0 conn(0x7fbb380a2750 legacy=0x7fbb380999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 1 -- 172.21.15.37:0/705681143 >> v1:172.21.15.37:6789/0 conn(0x7fbb38071880 legacy=0x7fbb3809a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 10 monclient: _finish_hunting 0 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 1 monclient: found mon.noname-a 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 1 -- 172.21.15.37:0/705681143 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032530591 0 0) 0x7fbb380a2b30 con 0x7fbb38071880 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 1 -- 172.21.15.37:0/705681143 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3809a800 con 0x7fbb380a31e0 2024-09-16T07:19:02.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3ed4c700 1 --2- 172.21.15.37:0/705681143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb380a31e0 0x7fbb3813a060 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbb3809d220 tx=0x7fbb3000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e8f18edcf2bcfc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 1 -- 172.21.15.37:0/705681143 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3000f070 con 0x7fbb380a31e0 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 10 monclient: dump: 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.566 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 10 monclient: _finish_auth 0 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 10 monclient: _check_auth_tickets 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 5 monclient: authenticate success, global_id 4389 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 1 -- 172.21.15.37:0/705681143 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3000a2e0 con 0x7fbb380a31e0 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 4 set_mon_vals no callback set 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 1 -- 172.21.15.37:0/705681143 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30003c80 con 0x7fbb380a31e0 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:02.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3d549700 10 monclient: dump: 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient: shutdown 2024-09-16T07:19:02.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 -- 172.21.15.37:0/705681143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb380a31e0 msgr2=0x7fbb3813a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 --2- 172.21.15.37:0/705681143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb380a31e0 0x7fbb3813a060 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbb3809d220 tx=0x7fbb3000c330 comp rx=0 tx=0).stop 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 -- 172.21.15.37:0/705681143 shutdown_connections 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 --2- 172.21.15.37:0/705681143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb380a31e0 0x7fbb3813a060 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 -- 172.21.15.37:0/705681143 >> 172.21.15.37:0/705681143 conn(0x7fbb380a3ea0 msgr2=0x7fbb380a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 -- 172.21.15.37:0/705681143 shutdown_connections 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 1 -- 172.21.15.37:0/705681143 wait complete. 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient: build_initial_monmap 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.560+0000 7fbb3f54d700 10 monclient: monmap: 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:02.569 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 1 Processor -- start 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 1 -- start start 2024-09-16T07:19:02.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient: init 2024-09-16T07:19:02.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:02.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:19:02.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb38071880 0x7fbb381c4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient(hunting): picked mon.noname-a con 0x7fbb38071880 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb380a2750 0x7fbb381c1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient(hunting): picked mon.noname-b con 0x7fbb380a2750 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb380a31e0 0x7fbb381c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient(hunting): picked mon.noname-c con 0x7fbb380a31e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380708c0 con 0x7fbb38071880 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb381c57e0 con 0x7fbb380a2750 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380724c0 con 0x7fbb380a31e0 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient(hunting): authenticate will time out at 863.053604s 2024-09-16T07:19:02.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb38071880 0x7fbb381c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb380a31e0 0x7fbb381c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb38071880 0x7fbb381c4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52802/0 (socket says 172.21.15.37:52802) 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 1 -- 172.21.15.37:0/2435755547 learned_addr learned my addr 172.21.15.37:0/2435755547 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 10 monclient(hunting): get_auth_request con 0x7fbb38071880 auth_method 0 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 10 monclient(hunting): get_auth_request con 0x7fbb380a31e0 auth_method 0 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3dd4a700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb380a2750 0x7fbb381c1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3dd4a700 10 monclient(hunting): get_auth_request con 0x7fbb380a2750 auth_method 0 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3dd4a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3dd4a700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3dd4a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3dd4a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3dd4a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3dd4a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 1 -- 172.21.15.37:0/2435755547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb380a2750 msgr2=0x7fbb381c1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb380a2750 0x7fbb381c1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 1 -- 172.21.15.37:0/2435755547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb38071880 msgr2=0x7fbb381c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb38071880 0x7fbb381c4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 10 monclient: _finish_hunting 0 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 1 monclient: found mon.noname-c 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 1 -- 172.21.15.37:0/2435755547 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3000ab80 con 0x7fbb380a31e0 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3e54b700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb38071880 0x7fbb381c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3ed4c700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb380a31e0 0x7fbb381c7db0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbb3806c320 tx=0x7fbb30004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 1 -- 172.21.15.37:0/2435755547 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb30009c50 con 0x7fbb380a31e0 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 10 monclient: dump: 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 10 monclient: _finish_auth 0 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:02.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 1 -- 172.21.15.37:0/2435755547 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb30009db0 con 0x7fbb380a31e0 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 5 monclient: authenticate success, global_id 4337 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient: _renew_subs 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 1 -- 172.21.15.37:0/2435755547 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb381a6c80 con 0x7fbb380a31e0 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 1 -- 172.21.15.37:0/2435755547 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3001c630 con 0x7fbb380a31e0 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb2effd700 10 monclient: dump: 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.580 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient: _renew_subs 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3f54d700 1 -- 172.21.15.37:0/2435755547 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb381a9760 con 0x7fbb380a31e0 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.564+0000 7fbb3cd48700 4 set_mon_vals no callback set 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.568+0000 7fbb3f54d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.568+0000 7fbb3f54d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:02.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.568+0000 7fbb3f54d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:02.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.568+0000 7fbb3f54d700 1 -- 172.21.15.37:0/2435755547 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb380a0f60 con 0x7fbb380a31e0 2024-09-16T07:19:02.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.568+0000 7fbb2effd700 1 -- 172.21.15.37:0/2435755547 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fbb3001c790 con 0x7fbb380a31e0 2024-09-16T07:19:02.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.568+0000 7fbb2effd700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbb1c035600 0x7fbb1c037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.568+0000 7fbb2effd700 1 -- 172.21.15.37:0/2435755547 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4374+0+0 (secure 0 0 0) 0x7fbb30049750 con 0x7fbb380a31e0 2024-09-16T07:19:02.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.568+0000 7fbb3e54b700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbb1c035600 0x7fbb1c037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.568+0000 7fbb3e54b700 10 monclient: get_auth_request con 0x7fbb1c035600 auth_method 0 2024-09-16T07:19:02.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.568+0000 7fbb3e54b700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbb1c035600 0x7fbb1c037ac0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb34001d30 tx=0x7fbb34009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:02.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.572+0000 7fbb2effd700 1 -- 172.21.15.37:0/2435755547 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbb3001ca40 con 0x7fbb380a31e0 2024-09-16T07:19:02.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.572+0000 7fbb2effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:02.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.572+0000 7fbb2effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:02.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.744+0000 7fbb3f54d700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T07:19:02.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.744+0000 7fbb3f54d700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T07:19:02.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.744+0000 7fbb3f54d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:02.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.744+0000 7fbb3f54d700 1 -- 172.21.15.37:0/2435755547 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbb381a8f80 con 0x7fbb380a31e0 2024-09-16T07:19:02.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.744+0000 7fbb2effd700 1 -- 172.21.15.37:0/2435755547 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbb30017070 con 0x7fbb380a31e0 2024-09-16T07:19:02.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.744+0000 7fbb2effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T07:19:02.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.744+0000 7fbb2effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:02.745 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 8 2024-09-16T07:19:02.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 -- 172.21.15.37:0/2435755547 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbb1c035600 msgr2=0x7fbb1c037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbb1c035600 0x7fbb1c037ac0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb34001d30 tx=0x7fbb34009040 comp rx=0 tx=0).stop 2024-09-16T07:19:02.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 10 monclient: shutdown 2024-09-16T07:19:02.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:02.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 -- 172.21.15.37:0/2435755547 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb380a31e0 msgr2=0x7fbb381c7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb380a31e0 0x7fbb381c7db0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbb3806c320 tx=0x7fbb30004af0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 -- 172.21.15.37:0/2435755547 shutdown_connections 2024-09-16T07:19:02.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbb1c035600 0x7fbb1c037ac0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb380a31e0 0x7fbb381c7db0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb380a2750 0x7fbb381c1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 --2- 172.21.15.37:0/2435755547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb38071880 0x7fbb381c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 -- 172.21.15.37:0/2435755547 >> 172.21.15.37:0/2435755547 conn(0x7fbb380a3ea0 msgr2=0x7fbb38070d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:02.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 -- 172.21.15.37:0/2435755547 shutdown_connections 2024-09-16T07:19:02.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.748+0000 7fbb3f54d700 1 -- 172.21.15.37:0/2435755547 wait complete. 2024-09-16T07:19:02.764 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T07:19:02.764 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:19:02.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.876+0000 7fcec2bd9700 10 monclient: get_monmap_and_config 2024-09-16T07:19:02.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.876+0000 7fcec2bd9700 10 monclient: build_initial_monmap 2024-09-16T07:19:02.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.876+0000 7fcec2bd9700 10 monclient: monmap: 2024-09-16T07:19:02.879 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:02.879 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.879 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:02.883042+0000 2024-09-16T07:19:02.879 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:02.883042+0000 2024-09-16T07:19:02.879 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:02.879 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.879 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:02.879 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:02.880 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:02.880 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:02.880 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:02.880 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 1 Processor -- start 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 1 -- start start 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient: init 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcebc0a2750 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fcebc071880 0x7fcebc09a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient(hunting): picked mon.noname-a con 0x7fcebc071880 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcebc0a31e0 0x7fcebc139050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient(hunting): picked mon.noname-c con 0x7fcebc0a31e0 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:02.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc0a2b30 con 0x7fcebc0a2750 2024-09-16T07:19:02.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc09d0e0 con 0x7fcebc071880 2024-09-16T07:19:02.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc09bb40 con 0x7fcebc0a31e0 2024-09-16T07:19:02.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:02.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient(hunting): authenticate will time out at 863.369196s 2024-09-16T07:19:02.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec1bd7700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fcebc0a2750 0x7fcebc0999e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53420/0 (socket says 172.21.15.37:53420) 2024-09-16T07:19:02.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec1bd7700 1 -- 172.21.15.37:0/4071461827 learned_addr learned my addr 172.21.15.37:0/4071461827 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 1 --2- 172.21.15.37:0/4071461827 >> v2:172.21.15.37:3300/0 conn(0x7fcebc071880 0x7fcebc09a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 10 monclient(hunting): get_auth_request con 0x7fcebc071880 auth_method 0 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec23d8700 1 --2- 172.21.15.37:0/4071461827 >> v2:172.21.15.143:3300/0 conn(0x7fcebc0a31e0 0x7fcebc139050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec23d8700 10 monclient(hunting): get_auth_request con 0x7fcebc0a31e0 auth_method 0 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec23d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec23d8700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec23d8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 1 -- 172.21.15.37:0/4071461827 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308050443 0 0) 0x7fcebc0a2b30 con 0x7fcebc0a2750 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient(hunting): _negotiate 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient(hunting): my global_id is 4401 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 1 -- 172.21.15.37:0/4071461827 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea40057e0 con 0x7fcebc0a2750 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 10 monclient(hunting): handle_auth_done global_id 4398 payload 274 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 1 -- 172.21.15.37:0/4071461827 >> v2:172.21.15.143:3300/0 conn(0x7fcebc0a31e0 msgr2=0x7fcebc139050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 1 --2- 172.21.15.37:0/4071461827 >> v2:172.21.15.143:3300/0 conn(0x7fcebc0a31e0 0x7fcebc139050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 1 -- 172.21.15.37:0/4071461827 >> v1:172.21.15.37:6789/0 conn(0x7fcebc0a2750 legacy=0x7fcebc0999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 10 monclient: _finish_hunting 0 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 1 monclient: found mon.noname-a 2024-09-16T07:19:02.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 1 -- 172.21.15.37:0/4071461827 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc09c3e0 con 0x7fcebc071880 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec23d8700 1 --2- 172.21.15.37:0/4071461827 >> v2:172.21.15.143:3300/0 conn(0x7fcebc0a31e0 0x7fcebc139050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec13d6700 1 --2- 172.21.15.37:0/4071461827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc071880 0x7fcebc09a0f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcebc06b870 tx=0x7fceb0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9e8e15288ce162 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 1 -- 172.21.15.37:0/4071461827 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0011830 con 0x7fcebc071880 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient: dump: 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.888 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient: _finish_auth 0 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient: _check_auth_tickets 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 5 monclient: authenticate success, global_id 4398 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 1 -- 172.21.15.37:0/4071461827 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb0011990 con 0x7fcebc071880 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:02.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 4 set_mon_vals no callback set 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 1 -- 172.21.15.37:0/4071461827 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb0011d20 con 0x7fcebc071880 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec0bd5700 10 monclient: dump: 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.890 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.880+0000 7fcec2bd9700 10 monclient: shutdown 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 -- 172.21.15.37:0/4071461827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc071880 msgr2=0x7fcebc09a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 --2- 172.21.15.37:0/4071461827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc071880 0x7fcebc09a0f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcebc06b870 tx=0x7fceb0009b30 comp rx=0 tx=0).stop 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 -- 172.21.15.37:0/4071461827 shutdown_connections 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 --2- 172.21.15.37:0/4071461827 >> v2:172.21.15.143:3300/0 conn(0x7fcebc0a31e0 0x7fcebc139050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 --2- 172.21.15.37:0/4071461827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc071880 0x7fcebc09a0f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 -- 172.21.15.37:0/4071461827 >> 172.21.15.37:0/4071461827 conn(0x7fcebc0a3ea0 msgr2=0x7fcebc0a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 -- 172.21.15.37:0/4071461827 shutdown_connections 2024-09-16T07:19:02.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 -- 172.21.15.37:0/4071461827 wait complete. 2024-09-16T07:19:02.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 10 monclient: build_initial_monmap 2024-09-16T07:19:02.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 10 monclient: monmap: 2024-09-16T07:19:02.897 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:02.897 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:02.897 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:02.897 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:02.897 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:02.897 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:02.897 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:02.897 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 Processor -- start 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 -- start start 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 10 monclient: init 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcebc071880 0x7fcebc1c23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 10 monclient(hunting): picked mon.noname-b con 0x7fcebc071880 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc0a2750 0x7fcebc1bf950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 10 monclient(hunting): picked mon.noname-c con 0x7fcebc0a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:02.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc0a31e0 0x7fcebc1c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 10 monclient(hunting): picked mon.noname-a con 0x7fcebc0a31e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.884+0000 7fcec2bd9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc1a6a10 con 0x7fcebc0a31e0 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0707e0 con 0x7fcebc071880 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc0724c0 con 0x7fcebc0a2750 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 monclient(hunting): authenticate will time out at 863.376114s 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc0a2750 0x7fcebc1bf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcebc071880 0x7fcebc1c23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc0a31e0 0x7fcebc1c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc0a2750 0x7fcebc1bf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37970/0 (socket says 172.21.15.37:37970) 2024-09-16T07:19:02.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc0a31e0 0x7fcebc1c5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52832/0 (socket says 172.21.15.37:52832) 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 1 -- 172.21.15.37:0/3281032401 learned_addr learned my addr 172.21.15.37:0/3281032401 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 10 monclient(hunting): get_auth_request con 0x7fcebc0a2750 auth_method 0 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcebc071880 0x7fcebc1c23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.37:39370/0 (socket says 172.21.15.37:39370) 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 10 monclient(hunting): get_auth_request con 0x7fcebc0a31e0 auth_method 0 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 10 monclient(hunting): get_auth_request con 0x7fcebc071880 auth_method 0 2024-09-16T07:19:02.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:02.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 10 monclient(hunting): handle_auth_done global_id 4404 payload 824 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 1 -- 172.21.15.37:0/3281032401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc0a2750 msgr2=0x7fcebc1bf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc0a2750 0x7fcebc1bf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 1 -- 172.21.15.37:0/3281032401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcebc071880 msgr2=0x7fcebc1c23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcebc071880 0x7fcebc1c23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 10 monclient: _finish_hunting 0 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 1 monclient: found mon.noname-a 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 1 -- 172.21.15.37:0/3281032401 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb00094b0 con 0x7fcebc0a31e0 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec13d6700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc0a2750 0x7fcebc1bf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:19:02.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec23d8700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc0a31e0 0x7fcebc1c5900 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fceb400e810 tx=0x7fceb400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 1 -- 172.21.15.37:0/3281032401 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb400c070 con 0x7fcebc0a31e0 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 10 monclient: dump: 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.903 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 10 monclient: _finish_auth 0 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 1 -- 172.21.15.37:0/3281032401 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb4004250 con 0x7fcebc0a31e0 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 1 -- 172.21.15.37:0/3281032401 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb40036c0 con 0x7fcebc0a31e0 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 10 monclient: dump: 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.904 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 5 monclient: authenticate success, global_id 4404 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcebbfff700 4 set_mon_vals no callback set 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 monclient: _renew_subs 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 1 -- 172.21.15.37:0/3281032401 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc1a47d0 con 0x7fcebc0a31e0 2024-09-16T07:19:02.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 monclient: _renew_subs 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 1 -- 172.21.15.37:0/3281032401 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc1a72b0 con 0x7fcebc0a31e0 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 1 -- 172.21.15.37:0/3281032401 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fceb4003820 con 0x7fcebc0a31e0 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcea8035600 0x7fcea8037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fceba7fc700 1 -- 172.21.15.37:0/3281032401 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4374+0+0 (secure 0 0 0) 0x7fceb4047620 con 0x7fcebc0a31e0 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec2bd9700 1 -- 172.21.15.37:0/3281032401 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc0a0c00 con 0x7fcebc0a31e0 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcea8035600 0x7fcea8037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.888+0000 7fcec1bd7700 10 monclient: get_auth_request con 0x7fcea8035600 auth_method 0 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.892+0000 7fcec1bd7700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcea8035600 0x7fcea8037ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fceac005cb0 tx=0x7fceac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.892+0000 7fceba7fc700 1 -- 172.21.15.37:0/3281032401 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fceb4013410 con 0x7fcebc0a31e0 2024-09-16T07:19:02.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.896+0000 7fceba7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:02.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:02.896+0000 7fceba7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:03.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.064+0000 7fcec2bd9700 1 -- 172.21.15.37:0/3281032401 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcebc09ab70 con 0x7fcea8035600 2024-09-16T07:19:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.068+0000 7fceba7fc700 1 -- 172.21.15.37:0/3281032401 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40838 (secure 0 0 0) 0x7fcebc09ab70 con 0x7fcea8035600 2024-09-16T07:19:03.070 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:19:03.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 -- 172.21.15.37:0/3281032401 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcea8035600 msgr2=0x7fcea8037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcea8035600 0x7fcea8037ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fceac005cb0 tx=0x7fceac009040 comp rx=0 tx=0).stop 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 10 monclient: shutdown 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 -- 172.21.15.37:0/3281032401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc0a31e0 msgr2=0x7fcebc1c5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc0a31e0 0x7fcebc1c5900 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fceb400e810 tx=0x7fceb400eb20 comp rx=0 tx=0).stop 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 -- 172.21.15.37:0/3281032401 shutdown_connections 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc0a31e0 0x7fcebc1c5900 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcea8035600 0x7fcea8037ac0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc0a2750 0x7fcebc1bf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 --2- 172.21.15.37:0/3281032401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcebc071880 0x7fcebc1c23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 -- 172.21.15.37:0/3281032401 >> 172.21.15.37:0/3281032401 conn(0x7fcebc0a3ea0 msgr2=0x7fcebc06ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 -- 172.21.15.37:0/3281032401 shutdown_connections 2024-09-16T07:19:03.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.072+0000 7fcec2bd9700 1 -- 172.21.15.37:0/3281032401 wait complete. 2024-09-16T07:19:03.073 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T07:19:03.086 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-16T07:19:02.445845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164332,"kb_used_data":2200,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066708,"statfs":{"total":579820584960,"available":579652308992,"internally_reserved":0,"allocated":2252800,"data_stored":1450804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":165927590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001729"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.593643+0000","last_change":"2024-09-16T07:18:57.472278+0000","last_active":"2024-09-16T07:19:00.593643+0000","last_peered":"2024-09-16T07:19:00.593643+0000","last_clean":"2024-09-16T07:19:00.593643+0000","last_became_active":"2024-09-16T07:18:55.463443+0000","last_became_peered":"2024-09-16T07:18:55.463443+0000","last_unstale":"2024-09-16T07:19:00.593643+0000","last_undegraded":"2024-09-16T07:19:00.593643+0000","last_fullsized":"2024-09-16T07:19:00.593643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:10.660959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059939100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.805232+0000","last_change":"2024-09-16T07:18:57.474462+0000","last_active":"2024-09-16T07:19:00.805232+0000","last_peered":"2024-09-16T07:19:00.805232+0000","last_clean":"2024-09-16T07:19:00.805232+0000","last_became_active":"2024-09-16T07:18:55.467864+0000","last_became_peered":"2024-09-16T07:18:55.467864+0000","last_unstale":"2024-09-16T07:19:00.805232+0000","last_undegraded":"2024-09-16T07:19:00.805232+0000","last_fullsized":"2024-09-16T07:19:00.805232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:02.070207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066039799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.593419+0000","last_change":"2024-09-16T07:18:57.472359+0000","last_active":"2024-09-16T07:19:00.593419+0000","last_peered":"2024-09-16T07:19:00.593419+0000","last_clean":"2024-09-16T07:19:00.593419+0000","last_became_active":"2024-09-16T07:18:55.471978+0000","last_became_peered":"2024-09-16T07:18:55.471978+0000","last_unstale":"2024-09-16T07:19:00.593419+0000","last_undegraded":"2024-09-16T07:19:00.593419+0000","last_fullsized":"2024-09-16T07:19:00.593419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:36.770184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077051799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.464442+0000","last_change":"2024-09-16T07:18:57.464442+0000","last_active":"2024-09-16T07:18:57.464442+0000","last_peered":"2024-09-16T07:18:57.464442+0000","last_clean":"2024-09-16T07:18:57.464442+0000","last_became_active":"2024-09-16T07:18:55.468584+0000","last_became_peered":"2024-09-16T07:18:55.468584+0000","last_unstale":"2024-09-16T07:18:57.464442+0000","last_undegraded":"2024-09-16T07:18:57.464442+0000","last_fullsized":"2024-09-16T07:18:57.464442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:20.380838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.597181+0000","last_change":"2024-09-16T07:18:57.470967+0000","last_active":"2024-09-16T07:19:00.597181+0000","last_peered":"2024-09-16T07:19:00.597181+0000","last_clean":"2024-09-16T07:19:00.597181+0000","last_became_active":"2024-09-16T07:18:55.472671+0000","last_became_peered":"2024-09-16T07:18:55.472671+0000","last_unstale":"2024-09-16T07:19:00.597181+0000","last_undegraded":"2024-09-16T07:19:00.597181+0000","last_fullsized":"2024-09-16T07:19:00.597181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:13.933764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000840762,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.591465+0000","last_change":"2024-09-16T07:18:57.740277+0000","last_active":"2024-09-16T07:19:00.591465+0000","last_peered":"2024-09-16T07:19:00.591465+0000","last_clean":"2024-09-16T07:19:00.591465+0000","last_became_active":"2024-09-16T07:18:55.469151+0000","last_became_peered":"2024-09-16T07:18:55.469151+0000","last_unstale":"2024-09-16T07:19:00.591465+0000","last_undegraded":"2024-09-16T07:19:00.591465+0000","last_fullsized":"2024-09-16T07:19:00.591465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:34.482147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064042899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.931894+0000","last_change":"2024-09-16T07:18:57.743569+0000","last_active":"2024-09-16T07:19:00.931894+0000","last_peered":"2024-09-16T07:19:00.931894+0000","last_clean":"2024-09-16T07:19:00.931894+0000","last_became_active":"2024-09-16T07:18:55.469565+0000","last_became_peered":"2024-09-16T07:18:55.469565+0000","last_unstale":"2024-09-16T07:19:00.931894+0000","last_undegraded":"2024-09-16T07:19:00.931894+0000","last_fullsized":"2024-09-16T07:19:00.931894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:46.360051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077454399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.593409+0000","last_change":"2024-09-16T07:18:57.472930+0000","last_active":"2024-09-16T07:19:00.593409+0000","last_peered":"2024-09-16T07:19:00.593409+0000","last_clean":"2024-09-16T07:19:00.593409+0000","last_became_active":"2024-09-16T07:18:55.464362+0000","last_became_peered":"2024-09-16T07:18:55.464362+0000","last_unstale":"2024-09-16T07:19:00.593409+0000","last_undegraded":"2024-09-16T07:19:00.593409+0000","last_fullsized":"2024-09-16T07:19:00.593409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:51.926284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212957,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.471579+0000","last_change":"2024-09-16T07:18:49.375792+0000","last_active":"2024-09-16T07:18:57.471579+0000","last_peered":"2024-09-16T07:18:57.471579+0000","last_clean":"2024-09-16T07:18:57.471579+0000","last_became_active":"2024-09-16T07:18:49.373551+0000","last_became_peered":"2024-09-16T07:18:49.373551+0000","last_unstale":"2024-09-16T07:18:57.471579+0000","last_undegraded":"2024-09-16T07:18:57.471579+0000","last_fullsized":"2024-09-16T07:18:57.471579+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_clean_scrub_stamp":"2024-09-16T07:18:47.386541+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:31.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":67019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":67019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":67038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]}]},{"osd":4,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":62436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":733184,"data_stored":595947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":720896,"data_stored":591345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:19:03.088 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.200+0000 7f50b56f8700 10 monclient: get_monmap_and_config 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.200+0000 7f50b56f8700 10 monclient: build_initial_monmap 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.200+0000 7f50b56f8700 10 monclient: monmap: 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:03.207220+0000 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:03.207220+0000 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:03.203 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:03.204 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:03.204 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 Processor -- start 2024-09-16T07:19:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 -- start start 2024-09-16T07:19:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient: init 2024-09-16T07:19:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T07:19:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f50b01334e0 0x7f50b01338c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient(hunting): picked mon.noname-c con 0x7f50b01334e0 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50b009cd00 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f50b009fad0 0x7f50b013b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient(hunting): picked mon.noname-a con 0x7f50b009fad0 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b009a030 con 0x7f50b009cd00 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b0071200 con 0x7f50b009fad0 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b009a230 con 0x7f50b01334e0 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient(hunting): authenticate will time out at 863.693035s 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f50b009fad0 0x7f50b013b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50aeffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f50b01334e0 0x7f50b01338c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f50b009fad0 0x7f50b013b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52840/0 (socket says 172.21.15.37:52840) 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 1 -- 172.21.15.37:0/2722546205 learned_addr learned my addr 172.21.15.37:0/2722546205 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 10 monclient(hunting): get_auth_request con 0x7f50b009fad0 auth_method 0 2024-09-16T07:19:03.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50a67fc700 1 --1- 172.21.15.37:0/2722546205 >> v1:172.21.15.143:6789/0 conn(0x7f50b009cd00 0x7f50b009f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.37:34662/0 (socket says 172.21.15.37:34662) 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50aeffd700 10 monclient(hunting): get_auth_request con 0x7f50b01334e0 auth_method 0 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50aeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50aeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50aeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50aeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 10 monclient(hunting): handle_auth_done global_id 4407 payload 274 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 1 -- 172.21.15.37:0/2722546205 >> v2:172.21.15.143:3300/0 conn(0x7f50b01334e0 msgr2=0x7f50b01338c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 1 --2- 172.21.15.37:0/2722546205 >> v2:172.21.15.143:3300/0 conn(0x7f50b01334e0 0x7f50b01338c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 1 -- 172.21.15.37:0/2722546205 >> v1:172.21.15.143:6789/0 conn(0x7f50b009cd00 legacy=0x7f50b009f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 1 monclient: found mon.noname-a 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 1 -- 172.21.15.37:0/2722546205 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b009d6f0 con 0x7f50b009fad0 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50af7fe700 1 --2- 172.21.15.37:0/2722546205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009fad0 0x7f50b013b3f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f50b006c0a0 tx=0x7f50a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c105de07cc6696d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 1 -- 172.21.15.37:0/2722546205 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a800e070 con 0x7f50b009fad0 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:03.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 10 monclient: dump: 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 10 monclient: _finish_auth 0 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:03.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 5 monclient: authenticate success, global_id 4407 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 1 -- 172.21.15.37:0/2722546205 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a800b910 con 0x7f50b009fad0 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 4 set_mon_vals no callback set 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 1 -- 172.21.15.37:0/2722546205 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a800bcc0 con 0x7f50b009fad0 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50ae7fc700 10 monclient: dump: 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.212 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 10 monclient: shutdown 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 -- 172.21.15.37:0/2722546205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009fad0 msgr2=0x7f50b013b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 --2- 172.21.15.37:0/2722546205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009fad0 0x7f50b013b3f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f50b006c0a0 tx=0x7f50a800b330 comp rx=0 tx=0).stop 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 -- 172.21.15.37:0/2722546205 shutdown_connections 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 --2- 172.21.15.37:0/2722546205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009fad0 0x7f50b013b3f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 --2- 172.21.15.37:0/2722546205 >> v2:172.21.15.143:3300/0 conn(0x7f50b01334e0 0x7f50b01338c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 -- 172.21.15.37:0/2722546205 >> 172.21.15.37:0/2722546205 conn(0x7f50b00a04b0 msgr2=0x7f50b00a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 -- 172.21.15.37:0/2722546205 shutdown_connections 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.204+0000 7f50b56f8700 1 -- 172.21.15.37:0/2722546205 wait complete. 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient: build_initial_monmap 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient: monmap: 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:03.214 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 1 Processor -- start 2024-09-16T07:19:03.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 1 -- start start 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient: init 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009cd00 0x7f50b01c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient(hunting): picked mon.noname-a con 0x7f50b009cd00 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50b009fad0 0x7f50b01c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient(hunting): picked mon.noname-c con 0x7f50b009fad0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50b01334e0 0x7f50b011d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient(hunting): picked mon.noname-b con 0x7f50b01334e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b009c1a0 con 0x7f50b009cd00 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b01218b0 con 0x7f50b01334e0 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00a33b0 con 0x7f50b009fad0 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient(hunting): authenticate will time out at 863.697418s 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009cd00 0x7f50b01c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a67fc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50b009fad0 0x7f50b01c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009cd00 0x7f50b01c61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52848/0 (socket says 172.21.15.37:52848) 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 1 -- 172.21.15.37:0/801713150 learned_addr learned my addr 172.21.15.37:0/801713150 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 10 monclient(hunting): get_auth_request con 0x7f50b009cd00 auth_method 0 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a67fc700 10 monclient(hunting): get_auth_request con 0x7f50b009fad0 auth_method 0 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50af7fe700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50b01334e0 0x7f50b011d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50af7fe700 10 monclient(hunting): get_auth_request con 0x7f50b01334e0 auth_method 0 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 10 monclient(hunting): handle_auth_done global_id 4410 payload 824 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 1 -- 172.21.15.37:0/801713150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50b009fad0 msgr2=0x7f50b01c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50b009fad0 0x7f50b01c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 1 -- 172.21.15.37:0/801713150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50b01334e0 msgr2=0x7f50b011d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50b01334e0 0x7f50b011d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 10 monclient: _finish_hunting 0 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 1 monclient: found mon.noname-a 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:03.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 1 -- 172.21.15.37:0/801713150 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a8009b30 con 0x7f50b009cd00 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50aeffd700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009cd00 0x7f50b01c61b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f50a0012fc0 tx=0x7f50a00100c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 1 -- 172.21.15.37:0/801713150 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a000dbc0 con 0x7f50b009cd00 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 10 monclient: dump: 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.223 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 10 monclient: _finish_auth 0 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 10 monclient: _check_auth_tickets 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 5 monclient: authenticate success, global_id 4410 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 1 -- 172.21.15.37:0/801713150 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a0013040 con 0x7f50b009cd00 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient: _renew_subs 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 1 -- 172.21.15.37:0/801713150 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b011fea0 con 0x7f50b009cd00 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 1 -- 172.21.15.37:0/801713150 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a0010b40 con 0x7f50b009cd00 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50ad7fa700 4 set_mon_vals no callback set 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 10 monclient: dump: 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient: _renew_subs 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50b56f8700 1 -- 172.21.15.37:0/801713150 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b011f550 con 0x7f50b009cd00 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.208+0000 7f50a7fff700 1 -- 172.21.15.37:0/801713150 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f50a0010ca0 con 0x7f50b009cd00 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.212+0000 7f50b56f8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.212+0000 7f50b56f8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.212+0000 7f50b56f8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.212+0000 7f50b56f8700 1 -- 172.21.15.37:0/801713150 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b01211a0 con 0x7f50b009cd00 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.212+0000 7f50a7fff700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5090035600 0x7f5090037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.212+0000 7f50a7fff700 1 -- 172.21.15.37:0/801713150 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4374+0+0 (secure 0 0 0) 0x7f50a004b990 con 0x7f50b009cd00 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.212+0000 7f50a67fc700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5090035600 0x7f5090037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.212+0000 7f50a67fc700 10 monclient: get_auth_request con 0x7f5090035600 auth_method 0 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.212+0000 7f50a67fc700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5090035600 0x7f5090037ac0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f50b0132f20 tx=0x7f5098007f30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.216+0000 7f50a7fff700 1 -- 172.21.15.37:0/801713150 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f50a0016750 con 0x7f50b009cd00 2024-09-16T07:19:03.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.216+0000 7f50a7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:03.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.216+0000 7f50a7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:03.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.380+0000 7f50b56f8700 1 -- 172.21.15.37:0/801713150 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50b0133e00 con 0x7f5090035600 2024-09-16T07:19:03.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50a7fff700 1 -- 172.21.15.37:0/801713150 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40838 (secure 0 0 0) 0x7f5098004b50 con 0x7f5090035600 2024-09-16T07:19:03.388 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:19:03.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 -- 172.21.15.37:0/801713150 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5090035600 msgr2=0x7f5090037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5090035600 0x7f5090037ac0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f50b0132f20 tx=0x7f5098007f30 comp rx=0 tx=0).stop 2024-09-16T07:19:03.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 10 monclient: shutdown 2024-09-16T07:19:03.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:03.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 -- 172.21.15.37:0/801713150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009cd00 msgr2=0x7f50b01c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009cd00 0x7f50b01c61b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f50a0012fc0 tx=0x7f50a00100c0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 -- 172.21.15.37:0/801713150 shutdown_connections 2024-09-16T07:19:03.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50b01334e0 0x7f50b011d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5090035600 0x7f5090037ac0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50b009fad0 0x7f50b01c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 --2- 172.21.15.37:0/801713150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b009cd00 0x7f50b01c61b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 -- 172.21.15.37:0/801713150 >> 172.21.15.37:0/801713150 conn(0x7f50b00a04b0 msgr2=0x7f50b00a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 -- 172.21.15.37:0/801713150 shutdown_connections 2024-09-16T07:19:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.388+0000 7f50b56f8700 1 -- 172.21.15.37:0/801713150 wait complete. 2024-09-16T07:19:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T07:19:03.403 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-16T07:19:02.445845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164332,"kb_used_data":2200,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066708,"statfs":{"total":579820584960,"available":579652308992,"internally_reserved":0,"allocated":2252800,"data_stored":1450804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":165927590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001729"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.593643+0000","last_change":"2024-09-16T07:18:57.472278+0000","last_active":"2024-09-16T07:19:00.593643+0000","last_peered":"2024-09-16T07:19:00.593643+0000","last_clean":"2024-09-16T07:19:00.593643+0000","last_became_active":"2024-09-16T07:18:55.463443+0000","last_became_peered":"2024-09-16T07:18:55.463443+0000","last_unstale":"2024-09-16T07:19:00.593643+0000","last_undegraded":"2024-09-16T07:19:00.593643+0000","last_fullsized":"2024-09-16T07:19:00.593643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:10.660959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059939100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.805232+0000","last_change":"2024-09-16T07:18:57.474462+0000","last_active":"2024-09-16T07:19:00.805232+0000","last_peered":"2024-09-16T07:19:00.805232+0000","last_clean":"2024-09-16T07:19:00.805232+0000","last_became_active":"2024-09-16T07:18:55.467864+0000","last_became_peered":"2024-09-16T07:18:55.467864+0000","last_unstale":"2024-09-16T07:19:00.805232+0000","last_undegraded":"2024-09-16T07:19:00.805232+0000","last_fullsized":"2024-09-16T07:19:00.805232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:02.070207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066039799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.593419+0000","last_change":"2024-09-16T07:18:57.472359+0000","last_active":"2024-09-16T07:19:00.593419+0000","last_peered":"2024-09-16T07:19:00.593419+0000","last_clean":"2024-09-16T07:19:00.593419+0000","last_became_active":"2024-09-16T07:18:55.471978+0000","last_became_peered":"2024-09-16T07:18:55.471978+0000","last_unstale":"2024-09-16T07:19:00.593419+0000","last_undegraded":"2024-09-16T07:19:00.593419+0000","last_fullsized":"2024-09-16T07:19:00.593419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:36.770184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077051799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.464442+0000","last_change":"2024-09-16T07:18:57.464442+0000","last_active":"2024-09-16T07:18:57.464442+0000","last_peered":"2024-09-16T07:18:57.464442+0000","last_clean":"2024-09-16T07:18:57.464442+0000","last_became_active":"2024-09-16T07:18:55.468584+0000","last_became_peered":"2024-09-16T07:18:55.468584+0000","last_unstale":"2024-09-16T07:18:57.464442+0000","last_undegraded":"2024-09-16T07:18:57.464442+0000","last_fullsized":"2024-09-16T07:18:57.464442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:20.380838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.597181+0000","last_change":"2024-09-16T07:18:57.470967+0000","last_active":"2024-09-16T07:19:00.597181+0000","last_peered":"2024-09-16T07:19:00.597181+0000","last_clean":"2024-09-16T07:19:00.597181+0000","last_became_active":"2024-09-16T07:18:55.472671+0000","last_became_peered":"2024-09-16T07:18:55.472671+0000","last_unstale":"2024-09-16T07:19:00.597181+0000","last_undegraded":"2024-09-16T07:19:00.597181+0000","last_fullsized":"2024-09-16T07:19:00.597181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:13.933764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000840762,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.591465+0000","last_change":"2024-09-16T07:18:57.740277+0000","last_active":"2024-09-16T07:19:00.591465+0000","last_peered":"2024-09-16T07:19:00.591465+0000","last_clean":"2024-09-16T07:19:00.591465+0000","last_became_active":"2024-09-16T07:18:55.469151+0000","last_became_peered":"2024-09-16T07:18:55.469151+0000","last_unstale":"2024-09-16T07:19:00.591465+0000","last_undegraded":"2024-09-16T07:19:00.591465+0000","last_fullsized":"2024-09-16T07:19:00.591465+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:34.482147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064042899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.931894+0000","last_change":"2024-09-16T07:18:57.743569+0000","last_active":"2024-09-16T07:19:00.931894+0000","last_peered":"2024-09-16T07:19:00.931894+0000","last_clean":"2024-09-16T07:19:00.931894+0000","last_became_active":"2024-09-16T07:18:55.469565+0000","last_became_peered":"2024-09-16T07:18:55.469565+0000","last_unstale":"2024-09-16T07:19:00.931894+0000","last_undegraded":"2024-09-16T07:19:00.931894+0000","last_fullsized":"2024-09-16T07:19:00.931894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:46.360051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077454399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T07:19:00.593409+0000","last_change":"2024-09-16T07:18:57.472930+0000","last_active":"2024-09-16T07:19:00.593409+0000","last_peered":"2024-09-16T07:19:00.593409+0000","last_clean":"2024-09-16T07:19:00.593409+0000","last_became_active":"2024-09-16T07:18:55.464362+0000","last_became_peered":"2024-09-16T07:18:55.464362+0000","last_unstale":"2024-09-16T07:19:00.593409+0000","last_undegraded":"2024-09-16T07:19:00.593409+0000","last_fullsized":"2024-09-16T07:19:00.593409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:51.926284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212957,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T07:18:57.471579+0000","last_change":"2024-09-16T07:18:49.375792+0000","last_active":"2024-09-16T07:18:57.471579+0000","last_peered":"2024-09-16T07:18:57.471579+0000","last_clean":"2024-09-16T07:18:57.471579+0000","last_became_active":"2024-09-16T07:18:49.373551+0000","last_became_peered":"2024-09-16T07:18:49.373551+0000","last_unstale":"2024-09-16T07:18:57.471579+0000","last_undegraded":"2024-09-16T07:18:57.471579+0000","last_fullsized":"2024-09-16T07:18:57.471579+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_clean_scrub_stamp":"2024-09-16T07:18:47.386541+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:31.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":67019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":67019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":67038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]}]},{"osd":4,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":62436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":733184,"data_stored":595947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":720896,"data_stored":591345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:19:03.404 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T07:19:03.405 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T07:19:03.405 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T07:19:03.417 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T07:19:03.417 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T07:19:03.417 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:19:03.418 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T07:19:03.418 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726471143, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-16T07:19:03.418 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T07:19:03.418 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:19:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.524+0000 7f28f3fff700 10 monclient: get_monmap_and_config 2024-09-16T07:19:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.524+0000 7f28f3fff700 10 monclient: build_initial_monmap 2024-09-16T07:19:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.524+0000 7f28f3fff700 10 monclient: monmap: 2024-09-16T07:19:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:03.531085+0000 2024-09-16T07:19:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:03.531085+0000 2024-09-16T07:19:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:03.528 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 Processor -- start 2024-09-16T07:19:03.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 -- start start 2024-09-16T07:19:03.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient: init 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f28f40a5bb0 0x7f28f40a5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f28f40a5bb0 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f28f40a5780 0x7f28f4135220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f28f40a5780 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28f40a64d0 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f409a5a0 con 0x7f28f40a64d0 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f4099d00 con 0x7f28f40a5bb0 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f4099fe0 con 0x7f28f40a5780 2024-09-16T07:19:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient(hunting): authenticate will time out at 864.017019s 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f28f40a5bb0 0x7f28f40a5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f28f40a5bb0 0x7f28f40a5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52860/0 (socket says 172.21.15.37:52860) 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 1 -- 172.21.15.37:0/380661790 learned_addr learned my addr 172.21.15.37:0/380661790 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 10 monclient(hunting): get_auth_request con 0x7f28f40a5bb0 auth_method 0 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f27fc700 1 --2- 172.21.15.37:0/380661790 >> v2:172.21.15.143:3300/0 conn(0x7f28f40a5780 0x7f28f4135220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f27fc700 10 monclient(hunting): get_auth_request con 0x7f28f40a5780 auth_method 0 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 10 monclient(hunting): handle_auth_done global_id 4428 payload 274 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 1 -- 172.21.15.37:0/380661790 >> v2:172.21.15.143:3300/0 conn(0x7f28f40a5780 msgr2=0x7f28f4135220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 1 --2- 172.21.15.37:0/380661790 >> v2:172.21.15.143:3300/0 conn(0x7f28f40a5780 0x7f28f4135220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 1 -- 172.21.15.37:0/380661790 >> v1:172.21.15.143:6789/0 conn(0x7f28f40a64d0 legacy=0x7f28f4135760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 10 monclient: _finish_hunting 0 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 1 monclient: found mon.noname-a 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 1 -- 172.21.15.37:0/380661790 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f409a2c0 con 0x7f28f40a5bb0 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f2ffd700 1 --2- 172.21.15.37:0/380661790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a5bb0 0x7f28f40a5f90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f28dc009910 tx=0x7f28dc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6829dbbf87c8aed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 1 -- 172.21.15.37:0/380661790 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc00ba20 con 0x7f28f40a5bb0 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 10 monclient: dump: 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 10 monclient: _finish_auth 0 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 1 -- 172.21.15.37:0/380661790 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28dc00bb80 con 0x7f28f40a5bb0 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 1 -- 172.21.15.37:0/380661790 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc009e60 con 0x7f28f40a5bb0 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 5 monclient: authenticate success, global_id 4428 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 4 set_mon_vals no callback set 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f1ffb700 10 monclient: dump: 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 10 monclient: shutdown 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 -- 172.21.15.37:0/380661790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a5bb0 msgr2=0x7f28f40a5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 --2- 172.21.15.37:0/380661790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a5bb0 0x7f28f40a5f90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f28dc009910 tx=0x7f28dc009c20 comp rx=0 tx=0).stop 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 -- 172.21.15.37:0/380661790 shutdown_connections 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 --2- 172.21.15.37:0/380661790 >> v2:172.21.15.143:3300/0 conn(0x7f28f40a5780 0x7f28f4135220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 --2- 172.21.15.37:0/380661790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a5bb0 0x7f28f40a5f90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 -- 172.21.15.37:0/380661790 >> 172.21.15.37:0/380661790 conn(0x7f28f40a4480 msgr2=0x7f28f40a4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 -- 172.21.15.37:0/380661790 shutdown_connections 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.528+0000 7f28f3fff700 1 -- 172.21.15.37:0/380661790 wait complete. 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient: build_initial_monmap 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient: monmap: 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 1 Processor -- start 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 1 -- start start 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient: init 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28f40a5780 0x7f28f41ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f28f40a5780 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f28f40a5bb0 0x7f28f41ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f28f40a5bb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a64d0 0x7f28f41219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f28f40a64d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f4125350 con 0x7f28f40a64d0 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f4125910 con 0x7f28f40a5780 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f41235b0 con 0x7f28f40a5bb0 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f3fff700 10 monclient(hunting): authenticate will time out at 864.022421s 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a64d0 0x7f28f41219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f28f40a5bb0 0x7f28f41ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a64d0 0x7f28f41219b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52870/0 (socket says 172.21.15.37:52870) 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f28f40a5bb0 0x7f28f41ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37984/0 (socket says 172.21.15.37:37984) 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 1 -- 172.21.15.37:0/1533097310 learned_addr learned my addr 172.21.15.37:0/1533097310 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 10 monclient(hunting): get_auth_request con 0x7f28f40a64d0 auth_method 0 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 10 monclient(hunting): get_auth_request con 0x7f28f40a5bb0 auth_method 0 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f2ffd700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28f40a5780 0x7f28f41ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f2ffd700 10 monclient(hunting): get_auth_request con 0x7f28f40a5780 auth_method 0 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 1 -- 172.21.15.37:0/1533097310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28f40a5780 msgr2=0x7f28f41ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28f40a5780 0x7f28f41ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 1 -- 172.21.15.37:0/1533097310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a64d0 msgr2=0x7f28f41219b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a64d0 0x7f28f41219b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 10 monclient: _finish_hunting 0 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 1 monclient: found mon.noname-c 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 1 -- 172.21.15.37:0/1533097310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc009570 con 0x7f28f40a5bb0 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f37fe700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a64d0 0x7f28f41219b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.532+0000 7f28f27fc700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f28f40a5bb0 0x7f28f41ca8c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f28e400e7e0 tx=0x7f28e400eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 1 -- 172.21.15.37:0/1533097310 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4004230 con 0x7f28f40a5bb0 2024-09-16T07:19:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 10 monclient: dump: 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 10 monclient: _finish_auth 0 2024-09-16T07:19:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 1 -- 172.21.15.37:0/1533097310 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e4004390 con 0x7f28f40a5bb0 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 5 monclient: authenticate success, global_id 4355 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 10 monclient: _renew_subs 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 1 -- 172.21.15.37:0/1533097310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f40a3f60 con 0x7f28f40a5bb0 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 1 -- 172.21.15.37:0/1533097310 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e40036c0 con 0x7f28f40a5bb0 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 10 monclient: dump: 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f17fa700 4 set_mon_vals no callback set 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 10 monclient: _renew_subs 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 1 -- 172.21.15.37:0/1533097310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f4125070 con 0x7f28f40a5bb0 2024-09-16T07:19:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 1 -- 172.21.15.37:0/1533097310 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f28e4003820 con 0x7f28f40a5bb0 2024-09-16T07:19:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f3fff700 1 -- 172.21.15.37:0/1533097310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f40a0210 con 0x7f28f40a5bb0 2024-09-16T07:19:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f28d4035630 0x7f28d4037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28eb7fe700 1 -- 172.21.15.37:0/1533097310 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 4374+0+0 (secure 0 0 0) 0x7f28e4047400 con 0x7f28f40a5bb0 2024-09-16T07:19:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f2ffd700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f28d4035630 0x7f28d4037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f2ffd700 10 monclient: get_auth_request con 0x7f28d4035630 auth_method 0 2024-09-16T07:19:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.536+0000 7f28f2ffd700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f28d4035630 0x7f28d4037af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f28dc000c00 tx=0x7f28dc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.544+0000 7f28eb7fe700 1 -- 172.21.15.37:0/1533097310 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f28e401b960 con 0x7f28f40a5bb0 2024-09-16T07:19:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.544+0000 7f28eb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.544+0000 7f28eb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:19:03.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:19:03.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:03.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 -- 172.21.15.37:0/1533097310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f28f40a5f90 con 0x7f28f40a5bb0 2024-09-16T07:19:03.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28eb7fe700 1 -- 172.21.15.37:0/1533097310 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f28e401b780 con 0x7f28f40a5bb0 2024-09-16T07:19:03.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28eb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:19:03.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28eb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:03.789 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:19:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 -- 172.21.15.37:0/1533097310 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f28d4035630 msgr2=0x7f28d4037af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f28d4035630 0x7f28d4037af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f28dc000c00 tx=0x7f28dc008030 comp rx=0 tx=0).stop 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 10 monclient: shutdown 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 -- 172.21.15.37:0/1533097310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f28f40a5bb0 msgr2=0x7f28f41ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f28f40a5bb0 0x7f28f41ca8c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f28e400e7e0 tx=0x7f28e400eaf0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 -- 172.21.15.37:0/1533097310 shutdown_connections 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28f40a64d0 0x7f28f41219b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f28d4035630 0x7f28d4037af0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f28f40a5bb0 0x7f28f41ca8c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 --2- 172.21.15.37:0/1533097310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28f40a5780 0x7f28f41ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 -- 172.21.15.37:0/1533097310 >> 172.21.15.37:0/1533097310 conn(0x7f28f40a4480 msgr2=0x7f28f40a0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 -- 172.21.15.37:0/1533097310 shutdown_connections 2024-09-16T07:19:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.788+0000 7f28f3fff700 1 -- 172.21.15.37:0/1533097310 wait complete. 2024-09-16T07:19:03.802 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T07:19:03.803 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T07:19:03.803 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T07:19:03.803 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T07:19:03.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.916+0000 7f16455d5700 10 monclient: get_monmap_and_config 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.916+0000 7f16455d5700 10 monclient: build_initial_monmap 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.916+0000 7f16455d5700 10 monclient: monmap: 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:03.923002+0000 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:03.923002+0000 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:03.920 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 Processor -- start 2024-09-16T07:19:03.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 -- start start 2024-09-16T07:19:03.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient: init 2024-09-16T07:19:03.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:03.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T07:19:03.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 0x7f16400a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient(hunting): picked mon.noname-b con 0x7f16400a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:03.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16400a3070 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:03.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1640133730 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:03.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1640099be0 con 0x7f1640133730 2024-09-16T07:19:03.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f164009de40 con 0x7f16400a3070 2024-09-16T07:19:03.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164009d530 con 0x7f16400a2750 2024-09-16T07:19:03.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:03.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient(hunting): authenticate will time out at 864.408811s 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 0x7f16400a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f1644dd4700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1640133730 0x7f164013b310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53430/0 (socket says 172.21.15.37:53430) 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 0x7f16400a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37994/0 (socket says 172.21.15.37:37994) 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f1644dd4700 1 -- 172.21.15.37:0/2982478546 learned_addr learned my addr 172.21.15.37:0/2982478546 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 10 monclient(hunting): get_auth_request con 0x7f16400a2750 auth_method 0 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2024-09-16T07:19:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 1 -- 172.21.15.37:0/2982478546 >> v1:172.21.15.143:6789/0 conn(0x7f16400a3070 legacy=0x7f1640133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 1 -- 172.21.15.37:0/2982478546 >> v1:172.21.15.37:6789/0 conn(0x7f1640133730 legacy=0x7f164013b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 1 -- 172.21.15.37:0/2982478546 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858118735 0 0) 0x7f1640099be0 con 0x7f1640133730 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 10 monclient: _finish_hunting 0 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 1 monclient: found mon.noname-b 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 1 -- 172.21.15.37:0/2982478546 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164009a020 con 0x7f16400a2750 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163ffff700 1 --2- 172.21.15.37:0/2982478546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 0x7f16400a2b30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1638009dc0 tx=0x7f163800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dfc9ce76f523f04 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 1 -- 172.21.15.37:0/2982478546 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163800e070 con 0x7f16400a2750 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 10 monclient: dump: 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 10 monclient: _finish_auth 0 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 5 monclient: authenticate success, global_id 4358 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 1 -- 172.21.15.37:0/2982478546 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163800bbd0 con 0x7f16400a2750 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 4 set_mon_vals no callback set 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 1 -- 172.21.15.37:0/2982478546 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638012430 con 0x7f16400a2750 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f163effd700 10 monclient: dump: 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 10 monclient: shutdown 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 -- 172.21.15.37:0/2982478546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 msgr2=0x7f16400a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 --2- 172.21.15.37:0/2982478546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 0x7f16400a2b30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1638009dc0 tx=0x7f163800b330 comp rx=0 tx=0).stop 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 -- 172.21.15.37:0/2982478546 shutdown_connections 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 --2- 172.21.15.37:0/2982478546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 0x7f16400a2b30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 -- 172.21.15.37:0/2982478546 >> 172.21.15.37:0/2982478546 conn(0x7f16400a3ea0 msgr2=0x7f16400a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:03.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 -- 172.21.15.37:0/2982478546 shutdown_connections 2024-09-16T07:19:03.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.920+0000 7f16455d5700 1 -- 172.21.15.37:0/2982478546 wait complete. 2024-09-16T07:19:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient: build_initial_monmap 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient: monmap: 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:03.935 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 1 Processor -- start 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 1 -- start start 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient: init 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 0x7f16401c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient(hunting): picked mon.noname-c con 0x7f16400a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16400a3070 0x7f1640132270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient(hunting): picked mon.noname-b con 0x7f16400a3070 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1640133730 0x7f16401327b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient(hunting): picked mon.noname-a con 0x7f1640133730 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164011d970 con 0x7f1640133730 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16401c84b0 con 0x7f16400a3070 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164006d020 con 0x7f16400a2750 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient(hunting): authenticate will time out at 864.413629s 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 0x7f16401c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 0x7f16401c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38004/0 (socket says 172.21.15.37:38004) 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163ffff700 1 -- 172.21.15.37:0/812705918 learned_addr learned my addr 172.21.15.37:0/812705918 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163ffff700 10 monclient(hunting): get_auth_request con 0x7f16400a2750 auth_method 0 2024-09-16T07:19:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1640133730 0x7f16401327b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 10 monclient(hunting): get_auth_request con 0x7f1640133730 auth_method 0 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 10 monclient(hunting): handle_auth_done global_id 4437 payload 824 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 1 -- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 msgr2=0x7f16401c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16400a2750 0x7f16401c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 1 -- 172.21.15.37:0/812705918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16400a3070 msgr2=0x7f1640132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16400a3070 0x7f1640132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 10 monclient: _finish_hunting 0 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 1 monclient: found mon.noname-a 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 1 -- 172.21.15.37:0/812705918 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1638009a70 con 0x7f1640133730 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f1644dd4700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1640133730 0x7f16401327b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1634000c00 tx=0x7f163400c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1634004ca0 con 0x7f1640133730 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 10 monclient: dump: 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 10 monclient: _finish_auth 0 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 10 monclient: _check_auth_tickets 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 5 monclient: authenticate success, global_id 4437 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient: _renew_subs 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1634004e20 con 0x7f1640133730 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 1 -- 172.21.15.37:0/812705918 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16400724c0 con 0x7f1640133730 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1634007310 con 0x7f1640133730 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:03.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163cff9700 10 monclient: dump: 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f163e7fc700 4 set_mon_vals no callback set 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient: _renew_subs 2024-09-16T07:19:03.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.924+0000 7f16455d5700 1 -- 172.21.15.37:0/812705918 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164011e1b0 con 0x7f1640133730 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f1634007470 con 0x7f1640133730 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f16455d5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f16455d5700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011bcb0 0x7f164011c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f16455d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f16455d5700 1 -- 172.21.15.37:0/812705918 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1640067530 con 0x7f164011bcb0 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f162c035600 0x7f162c037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f1634047790 con 0x7f1640133730 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011bcb0 0x7f164011c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 10 monclient: get_auth_request con 0x7f164011bcb0 auth_method 0 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2024-09-16T07:19:03.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163f7fe700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f162c035600 0x7f162c037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 1 -- 172.21.15.37:0/812705918 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16400a3450 con 0x7f164011bcb0 2024-09-16T07:19:03.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163f7fe700 10 monclient: get_auth_request con 0x7f162c035600 auth_method 0 2024-09-16T07:19:03.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163ffff700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011bcb0 0x7f164011c090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1638009d90 tx=0x7f1638004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638012c00 con 0x7f164011bcb0 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 10 monclient: dump: 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.945 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.946 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.946 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163f7fe700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f162c035600 0x7f162c037ac0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1630000f60 tx=0x7f1630007a60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f1638012d60 con 0x7f164011bcb0 2024-09-16T07:19:03.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:03.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:03.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011bcb0 msgr2=0x7f164011c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011bcb0 0x7f164011c090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1638009d90 tx=0x7f1638004af0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011bcb0 msgr2=0x7f164011c090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:19:03.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.928+0000 7f163cff9700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011bcb0 0x7f164011c090 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1638009d90 tx=0x7f1638004af0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.936+0000 7f16455d5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.936+0000 7f16455d5700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011c5d0 0x7f164006e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.936+0000 7f16455d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.936+0000 7f16455d5700 1 -- 172.21.15.37:0/812705918 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164011f3f0 con 0x7f164011c5d0 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.936+0000 7f163ffff700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011c5d0 0x7f164006e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.936+0000 7f163ffff700 10 monclient: get_auth_request con 0x7f164011c5d0 auth_method 0 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.936+0000 7f163ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.936+0000 7f163ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:03.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163ffff700 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163ffff700 1 -- 172.21.15.37:0/812705918 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f164009cd90 con 0x7f164011c5d0 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163ffff700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011c5d0 0x7f164006e290 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1638024000 tx=0x7f1638012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163802c2b0 con 0x7f164011c5d0 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 10 monclient: dump: 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.949 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4622 (secure 0 0 0) 0x7f163802c890 con 0x7f164011c5d0 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011c5d0 msgr2=0x7f164006e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011c5d0 0x7f164006e290 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1638024000 tx=0x7f1638012000 comp rx=0 tx=0).stop 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 1 -- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011c5d0 msgr2=0x7f164006e290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:19:03.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f163cff9700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f164011c5d0 0x7f164006e290 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1638024000 tx=0x7f1638012000 comp rx=0 tx=0).stop 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 12, 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:03.951 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 27, 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:03.952 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:03.953 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:19:03.954 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:03.955 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:03.956 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.957 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:03.958 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:03.959 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.960 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:03.961 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:03.962 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:19:03.963 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 1 -- 172.21.15.37:0/812705918 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f162c035600 msgr2=0x7f162c037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f162c035600 0x7f162c037ac0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1630000f60 tx=0x7f1630007a60 comp rx=0 tx=0).stop 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 10 monclient: shutdown 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f1644dd4700 1 -- 172.21.15.37:0/812705918 reap_dead start 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 1 -- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1640133730 msgr2=0x7f16401327b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1640133730 0x7f16401327b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1634000c00 tx=0x7f163400c710 comp rx=0 tx=0).stop 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 1 -- 172.21.15.37:0/812705918 shutdown_connections 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 1 --2- 172.21.15.37:0/812705918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1640133730 0x7f16401327b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 1 -- 172.21.15.37:0/812705918 >> 172.21.15.37:0/812705918 conn(0x7f16400a3ea0 msgr2=0x7f16401b88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:03.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 1 -- 172.21.15.37:0/812705918 shutdown_connections 2024-09-16T07:19:03.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:03.940+0000 7f16455d5700 1 -- 172.21.15.37:0/812705918 wait complete. 2024-09-16T07:19:03.966 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:19:03.966 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T07:19:03.966 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T07:19:04.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient: get_monmap_and_config 2024-09-16T07:19:04.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient: build_initial_monmap 2024-09-16T07:19:04.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient: monmap: 2024-09-16T07:19:04.084 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:04.084 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.084 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:04.087719+0000 2024-09-16T07:19:04.084 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:04.087719+0000 2024-09-16T07:19:04.084 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:04.084 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.084 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:04.085 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:04.085 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:04.085 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:04.085 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:04.085 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 1 Processor -- start 2024-09-16T07:19:04.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 1 -- start start 2024-09-16T07:19:04.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient: init 2024-09-16T07:19:04.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:04.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T07:19:04.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f578c0a5bb0 0x7f578c0a5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f578c0a5bb0 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5780 0x7f578c135220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f578c0a5780 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f578c0a64d0 0x7f578c135760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f578c0a64d0 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f578c0a16f0 con 0x7f578c0a64d0 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f578c09a5a0 con 0x7f578c0a5bb0 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578c099d00 con 0x7f578c0a5780 2024-09-16T07:19:04.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient(hunting): authenticate will time out at 864.574199s 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f578c0a64d0 0x7f578c135760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f5789d9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5780 0x7f578c135220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f578c0a64d0 0x7f578c135760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52908/0 (socket says 172.21.15.37:52908) 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 1 -- 172.21.15.37:0/3317891087 learned_addr learned my addr 172.21.15.37:0/3317891087 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 10 monclient(hunting): get_auth_request con 0x7f578c0a64d0 auth_method 0 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f5789d9b700 10 monclient(hunting): get_auth_request con 0x7f578c0a5780 auth_method 0 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f5789d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f5789d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f5789d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578a59c700 1 --2- 172.21.15.37:0/3317891087 >> v2:172.21.15.143:3300/0 conn(0x7f578c0a5bb0 0x7f578c0a5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f5789d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f5789d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f5789d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578a59c700 10 monclient(hunting): get_auth_request con 0x7f578c0a5bb0 auth_method 0 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 10 monclient(hunting): handle_auth_done global_id 4446 payload 274 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 1 -- 172.21.15.37:0/3317891087 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5780 msgr2=0x7f578c135220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 1 --2- 172.21.15.37:0/3317891087 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5780 0x7f578c135220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 1 -- 172.21.15.37:0/3317891087 >> v2:172.21.15.143:3300/0 conn(0x7f578c0a5bb0 msgr2=0x7f578c0a5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 1 --2- 172.21.15.37:0/3317891087 >> v2:172.21.15.143:3300/0 conn(0x7f578c0a5bb0 0x7f578c0a5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 10 monclient: _finish_hunting 0 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 1 monclient: found mon.noname-a 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 1 -- 172.21.15.37:0/3317891087 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c099fe0 con 0x7f578c0a64d0 2024-09-16T07:19:04.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578ad9d700 1 --2- 172.21.15.37:0/3317891087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f578c0a64d0 0x7f578c135760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f578c0a8a40 tx=0x7f578400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a781203b8060b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 1 -- 172.21.15.37:0/3317891087 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578400e070 con 0x7f578c0a64d0 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 10 monclient: dump: 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.100 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 10 monclient: _finish_auth 0 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 10 monclient: _check_auth_tickets 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 5 monclient: authenticate success, global_id 4446 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 1 -- 172.21.15.37:0/3317891087 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578400b910 con 0x7f578c0a64d0 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578b59e700 4 set_mon_vals no callback set 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 1 -- 172.21.15.37:0/3317891087 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578400bcc0 con 0x7f578c0a64d0 2024-09-16T07:19:04.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.084+0000 7f578959a700 10 monclient: dump: 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient: shutdown 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 -- 172.21.15.37:0/3317891087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f578c0a64d0 msgr2=0x7f578c135760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 --2- 172.21.15.37:0/3317891087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f578c0a64d0 0x7f578c135760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f578c0a8a40 tx=0x7f578400b330 comp rx=0 tx=0).stop 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 -- 172.21.15.37:0/3317891087 shutdown_connections 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 --2- 172.21.15.37:0/3317891087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f578c0a64d0 0x7f578c135760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 --2- 172.21.15.37:0/3317891087 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5780 0x7f578c135220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 --2- 172.21.15.37:0/3317891087 >> v2:172.21.15.143:3300/0 conn(0x7f578c0a5bb0 0x7f578c0a5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 -- 172.21.15.37:0/3317891087 >> 172.21.15.37:0/3317891087 conn(0x7f578c0a4480 msgr2=0x7f578c0a4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 -- 172.21.15.37:0/3317891087 shutdown_connections 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 -- 172.21.15.37:0/3317891087 wait complete. 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient: build_initial_monmap 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient: monmap: 2024-09-16T07:19:04.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 Processor -- start 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 -- start start 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient: init 2024-09-16T07:19:04.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f578c0a5780 0x7f578c1ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f578c0a5780 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5bb0 0x7f578c1ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f578c0a5bb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f578c0a64d0 0x7f578c121990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f578c0a64d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c123590 con 0x7f578c0a5780 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c124160 con 0x7f578c0a64d0 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578c0a3ee0 con 0x7f578c0a5bb0 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578b59e700 10 monclient(hunting): authenticate will time out at 864.578981s 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f578c0a5780 0x7f578c1ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5bb0 0x7f578c1ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5bb0 0x7f578c1ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38014/0 (socket says 172.21.15.37:38014) 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f578c0a5780 0x7f578c1ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52920/0 (socket says 172.21.15.37:52920) 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 1 -- 172.21.15.37:0/1050131273 learned_addr learned my addr 172.21.15.37:0/1050131273 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 10 monclient(hunting): get_auth_request con 0x7f578c0a5bb0 auth_method 0 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578a59c700 10 monclient(hunting): get_auth_request con 0x7f578c0a5780 auth_method 0 2024-09-16T07:19:04.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578ad9d700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f578c0a64d0 0x7f578c121990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578ad9d700 10 monclient(hunting): get_auth_request con 0x7f578c0a64d0 auth_method 0 2024-09-16T07:19:04.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 1 -- 172.21.15.37:0/1050131273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f578c0a64d0 msgr2=0x7f578c121990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f578c0a64d0 0x7f578c121990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 1 -- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f578c0a5780 msgr2=0x7f578c1ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f578c0a5780 0x7f578c1ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 10 monclient: _finish_hunting 0 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 1 monclient: found mon.noname-c 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 1 -- 172.21.15.37:0/1050131273 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5784009b30 con 0x7f578c0a5bb0 2024-09-16T07:19:04.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f578a59c700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f578c0a5780 0x7f578c1ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.088+0000 7f5789d9b700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5bb0 0x7f578c1ca8c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f578000e910 tx=0x7f578000ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578000ee60 con 0x7f578c0a5bb0 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: dump: 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.109 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.110 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.110 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: _finish_auth 0 2024-09-16T07:19:04.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:04.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:04.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 5 monclient: authenticate success, global_id 4367 2024-09-16T07:19:04.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578000c850 con 0x7f578c0a5bb0 2024-09-16T07:19:04.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 10 monclient: _renew_subs 2024-09-16T07:19:04.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:04.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 1 -- 172.21.15.37:0/1050131273 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578c0a0050 con 0x7f578c0a5bb0 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 10 monclient: _renew_subs 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f5788d99700 4 set_mon_vals no callback set 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 1 -- 172.21.15.37:0/1050131273 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578c125610 con 0x7f578c0a5bb0 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578000cbc0 con 0x7f578c0a5bb0 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: dump: 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.111 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f578000cd20 con 0x7f578c0a5bb0 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c121ed0 0x7f578c1222b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578b59e700 1 -- 172.21.15.37:0/1050131273 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578c067380 con 0x7f578c121ed0 2024-09-16T07:19:04.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f576c035600 0x7f576c037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c121ed0 0x7f578c1222b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f5780048120 con 0x7f578c0a5bb0 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 10 monclient: get_auth_request con 0x7f578c121ed0 auth_method 0 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578ad9d700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f576c035600 0x7f576c037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578ad9d700 10 monclient: get_auth_request con 0x7f576c035600 auth_method 0 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-09-16T07:19:04.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 1 -- 172.21.15.37:0/1050131273 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f578c09a380 con 0x7f578c121ed0 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578a59c700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c121ed0 0x7f578c1222b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f577c011d80 tx=0x7f577c00f520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c00fcc0 con 0x7f578c121ed0 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f578ad9d700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f576c035600 0x7f576c037ac0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f578400b370 tx=0x7f5784009000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: dump: 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.115 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f577c00fe20 con 0x7f578c121ed0 2024-09-16T07:19:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c121ed0 msgr2=0x7f578c1222b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c121ed0 0x7f578c1222b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f577c011d80 tx=0x7f577c00f520 comp rx=0 tx=0).stop 2024-09-16T07:19:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c121ed0 msgr2=0x7f578c1222b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:19:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.092+0000 7f577affd700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c121ed0 0x7f578c1222b0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f577c011d80 tx=0x7f577c00f520 comp rx=0 tx=0).stop 2024-09-16T07:19:04.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:19:04.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c1227f0 0x7f578c11dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 1 -- 172.21.15.37:0/1050131273 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578c125290 con 0x7f578c1227f0 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c1227f0 0x7f578c11dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 10 monclient: get_auth_request con 0x7f578c1227f0 auth_method 0 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 10 monclient(hunting): handle_auth_done global_id 4373 payload 824 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 1 -- 172.21.15.37:0/1050131273 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f578c135a60 con 0x7f578c1227f0 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578a59c700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c1227f0 0x7f578c11dfd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f577c01b000 tx=0x7f577c004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577c014640 con 0x7f578c1227f0 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 10 monclient: dump: 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4620 (secure 0 0 0) 0x7f577c0273a0 con 0x7f578c1227f0 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:19:04.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:04.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c1227f0 msgr2=0x7f578c11dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c1227f0 0x7f578c11dfd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f577c01b000 tx=0x7f577c004770 comp rx=0 tx=0).stop 2024-09-16T07:19:04.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 1 -- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c1227f0 msgr2=0x7f578c11dfd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:19:04.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f577affd700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c1227f0 0x7f578c11dfd0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f577c01b000 tx=0x7f577c004770 comp rx=0 tx=0).stop 2024-09-16T07:19:04.120 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:19:04.120 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:19:04.120 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:19:04.120 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:19:04.120 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 12, 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 27, 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:04.121 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:04.122 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:19:04.123 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:04.124 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:19:04.125 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.126 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.127 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:19:04.128 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.130 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:04.131 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:19:04.132 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:19:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 1 -- 172.21.15.37:0/1050131273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f576c035600 msgr2=0x7f576c037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f576c035600 0x7f576c037ac0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f578400b370 tx=0x7f5784009000 comp rx=0 tx=0).stop 2024-09-16T07:19:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 10 monclient: shutdown 2024-09-16T07:19:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 1 -- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5bb0 msgr2=0x7f578c1ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 1 --2- 172.21.15.37:0/1050131273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f578c0a5bb0 0x7f578c1ca8c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f578000e910 tx=0x7f578000ec20 comp rx=0 tx=0).stop 2024-09-16T07:19:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578ad9d700 1 -- 172.21.15.37:0/1050131273 reap_dead start 2024-09-16T07:19:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578ad9d700 1 -- 172.21.15.37:0/1050131273 reap_dead start 2024-09-16T07:19:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 1 -- 172.21.15.37:0/1050131273 shutdown_connections 2024-09-16T07:19:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 1 -- 172.21.15.37:0/1050131273 >> 172.21.15.37:0/1050131273 conn(0x7f578c0a4480 msgr2=0x7f578c09d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 1 -- 172.21.15.37:0/1050131273 shutdown_connections 2024-09-16T07:19:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.104+0000 7f578b59e700 1 -- 172.21.15.37:0/1050131273 wait complete. 2024-09-16T07:19:04.135 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:19:04.135 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T07:19:04.135 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T07:19:04.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient: get_monmap_and_config 2024-09-16T07:19:04.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient: build_initial_monmap 2024-09-16T07:19:04.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient: monmap: 2024-09-16T07:19:04.255 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:04.256 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.256 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:04.259652+0000 2024-09-16T07:19:04.256 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:04.259652+0000 2024-09-16T07:19:04.256 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:04.256 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.256 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:04.256 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:04.256 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:04.256 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:04.256 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:04.257 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 1 Processor -- start 2024-09-16T07:19:04.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 1 -- start start 2024-09-16T07:19:04.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient: init 2024-09-16T07:19:04.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:04.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T07:19:04.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7400a2750 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:04.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa7400a3240 0x7fa74009d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient(hunting): picked mon.noname-a con 0x7fa7400a3240 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:04.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa740137460 0x7fa740139850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient(hunting): picked mon.noname-c con 0x7fa740137460 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74009dac0 con 0x7fa7400a2750 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74009c6c0 con 0x7fa7400a3240 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74009c9a0 con 0x7fa740137460 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient(hunting): authenticate will time out at 864.745835s 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa7400a3240 0x7fa74009d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73ffff700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa7400a2750 0x7fa7400a2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53436/0 (socket says 172.21.15.37:53436) 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa7400a3240 0x7fa74009d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52932/0 (socket says 172.21.15.37:52932) 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73ffff700 1 -- 172.21.15.37:0/4290043914 learned_addr learned my addr 172.21.15.37:0/4290043914 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:04.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 10 monclient(hunting): get_auth_request con 0x7fa7400a3240 auth_method 0 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 10 monclient(hunting): handle_auth_done global_id 4452 payload 274 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 1 -- 172.21.15.37:0/4290043914 >> v2:172.21.15.143:3300/0 conn(0x7fa740137460 msgr2=0x7fa740139850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 1 --2- 172.21.15.37:0/4290043914 >> v2:172.21.15.143:3300/0 conn(0x7fa740137460 0x7fa740139850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 1 -- 172.21.15.37:0/4290043914 >> v1:172.21.15.37:6789/0 conn(0x7fa7400a2750 legacy=0x7fa7400a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:19:04.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 1 monclient: found mon.noname-a 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 1 -- 172.21.15.37:0/4290043914 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74009b130 con 0x7fa7400a3240 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73f7fe700 1 --2- 172.21.15.37:0/4290043914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7400a3240 0x7fa74009d580 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa72800baf0 tx=0x7fa72800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899c23a52d621564 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 1 -- 172.21.15.37:0/4290043914 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72800dcf0 con 0x7fa7400a3240 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 10 monclient: dump: 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.263 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 10 monclient: _finish_auth 0 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 1 -- 172.21.15.37:0/4290043914 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72800de50 con 0x7fa7400a3240 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 5 monclient: authenticate success, global_id 4452 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:04.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 1 -- 172.21.15.37:0/4290043914 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa728010690 con 0x7fa7400a3240 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa745260700 4 set_mon_vals no callback set 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.256+0000 7fa73effd700 10 monclient: dump: 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.265 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient: shutdown 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 -- 172.21.15.37:0/4290043914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7400a3240 msgr2=0x7fa74009d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 --2- 172.21.15.37:0/4290043914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7400a3240 0x7fa74009d580 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa72800baf0 tx=0x7fa72800be00 comp rx=0 tx=0).stop 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 -- 172.21.15.37:0/4290043914 shutdown_connections 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 --2- 172.21.15.37:0/4290043914 >> v2:172.21.15.143:3300/0 conn(0x7fa740137460 0x7fa740139850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 --2- 172.21.15.37:0/4290043914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7400a3240 0x7fa74009d580 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 -- 172.21.15.37:0/4290043914 >> 172.21.15.37:0/4290043914 conn(0x7fa7400a3ea0 msgr2=0x7fa7400a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 -- 172.21.15.37:0/4290043914 shutdown_connections 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 -- 172.21.15.37:0/4290043914 wait complete. 2024-09-16T07:19:04.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient: build_initial_monmap 2024-09-16T07:19:04.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient: monmap: 2024-09-16T07:19:04.267 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:04.267 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:04.267 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:04.267 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:04.267 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:04.267 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:04.267 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:04.267 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:04.268 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:04.268 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 Processor -- start 2024-09-16T07:19:04.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 -- start start 2024-09-16T07:19:04.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient: init 2024-09-16T07:19:04.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:04.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:19:04.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7400a2750 0x7fa7401ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient(hunting): picked mon.noname-b con 0x7fa7400a2750 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7400a3240 0x7fa7401cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient(hunting): picked mon.noname-a con 0x7fa7400a3240 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa740137460 0x7fa7401cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient(hunting): picked mon.noname-c con 0x7fa740137460 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400724c0 con 0x7fa7400a3240 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74011f630 con 0x7fa7400a2750 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74009f8e0 con 0x7fa740137460 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa745260700 10 monclient(hunting): authenticate will time out at 864.750709s 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7400a3240 0x7fa7401cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7400a3240 0x7fa7401cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52936/0 (socket says 172.21.15.37:52936) 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 1 -- 172.21.15.37:0/1769459624 learned_addr learned my addr 172.21.15.37:0/1769459624 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 10 monclient(hunting): get_auth_request con 0x7fa7400a3240 auth_method 0 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa744a5f700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa740137460 0x7fa7401cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa744a5f700 10 monclient(hunting): get_auth_request con 0x7fa740137460 auth_method 0 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa744a5f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa744a5f700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa744a5f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73ffff700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7400a2750 0x7fa7401ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 10 monclient(hunting): handle_auth_done global_id 4458 payload 824 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 1 -- 172.21.15.37:0/1769459624 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa740137460 msgr2=0x7fa7401cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa740137460 0x7fa7401cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 1 -- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7400a2750 msgr2=0x7fa7401ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7400a2750 0x7fa7401ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 1 monclient: found mon.noname-a 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 1 -- 172.21.15.37:0/1769459624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72800b750 con 0x7fa7400a3240 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa744a5f700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa740137460 0x7fa7401cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73ffff700 10 monclient: get_auth_request con 0x7fa7400a2750 auth_method 0 2024-09-16T07:19:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73ffff700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7400a2750 0x7fa7401ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T07:19:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73f7fe700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7400a3240 0x7fa7401cab70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa74006b870 tx=0x7fa728004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa728010470 con 0x7fa7400a3240 2024-09-16T07:19:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:19:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73cff9700 10 monclient: dump: 2024-09-16T07:19:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.278 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.278 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.278 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.278 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.278 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.278 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.279 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.279 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.279 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73cff9700 10 monclient: _finish_auth 0 2024-09-16T07:19:04.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73cff9700 10 monclient: _check_auth_tickets 2024-09-16T07:19:04.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.260+0000 7fa73cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:04.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 5 monclient: authenticate success, global_id 4458 2024-09-16T07:19:04.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 10 monclient: _renew_subs 2024-09-16T07:19:04.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:04.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 1 -- 172.21.15.37:0/1769459624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74009fe10 con 0x7fa7400a3240 2024-09-16T07:19:04.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7280105d0 con 0x7fa7400a3240 2024-09-16T07:19:04.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:04.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa728010960 con 0x7fa7400a3240 2024-09-16T07:19:04.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73e7fc700 4 set_mon_vals no callback set 2024-09-16T07:19:04.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:19:04.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 10 monclient: dump: 2024-09-16T07:19:04.283 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.283 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.283 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.284 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.285 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 10 monclient: _renew_subs 2024-09-16T07:19:04.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:19:04.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 1 -- 172.21.15.37:0/1769459624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74011f070 con 0x7fa7400a3240 2024-09-16T07:19:04.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740070ab0 0x7fa740134850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa745260700 1 -- 172.21.15.37:0/1769459624 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74011f7c0 con 0x7fa740070ab0 2024-09-16T07:19:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fa728019400 con 0x7fa7400a3240 2024-09-16T07:19:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740070ab0 0x7fa740134850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 10 monclient: get_auth_request con 0x7fa740070ab0 auth_method 0 2024-09-16T07:19:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa724035600 0x7fa724037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 10 monclient(hunting): handle_auth_done global_id 4438 payload 824 2024-09-16T07:19:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 1 -- 172.21.15.37:0/1769459624 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7400719d0 con 0x7fa740070ab0 2024-09-16T07:19:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4738+0+0 (secure 0 0 0) 0x7fa728048fe0 con 0x7fa7400a3240 2024-09-16T07:19:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73ffff700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740070ab0 0x7fa740134850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa730000f60 tx=0x7fa73000da80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa744a5f700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa724035600 0x7fa724037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73000b070 con 0x7fa740070ab0 2024-09-16T07:19:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:19:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 10 monclient: dump: 2024-09-16T07:19:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.292 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa744a5f700 10 monclient: get_auth_request con 0x7fa724035600 auth_method 0 2024-09-16T07:19:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fa730005d70 con 0x7fa740070ab0 2024-09-16T07:19:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740070ab0 msgr2=0x7fa740134850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.264+0000 7fa73cff9700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740070ab0 0x7fa740134850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa730000f60 tx=0x7fa73000da80 comp rx=0 tx=0).stop 2024-09-16T07:19:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.268+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740070ab0 msgr2=0x7fa740134850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:19:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.268+0000 7fa73cff9700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740070ab0 0x7fa740134850 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.268+0000 7fa744a5f700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa724035600 0x7fa724037ac0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa7400a1270 tx=0x7fa7380094f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:19:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740134d90 0x7fa7401351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 1 -- 172.21.15.37:0/1769459624 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740067530 con 0x7fa740134d90 2024-09-16T07:19:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740134d90 0x7fa7401351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 10 monclient: get_auth_request con 0x7fa740134d90 auth_method 0 2024-09-16T07:19:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 10 monclient(hunting): handle_auth_done global_id 4441 payload 824 2024-09-16T07:19:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 1 -- 172.21.15.37:0/1769459624 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa74011f230 con 0x7fa740134d90 2024-09-16T07:19:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73ffff700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740134d90 0x7fa7401351a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa730000f60 tx=0x7fa7300058a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa730020070 con 0x7fa740134d90 2024-09-16T07:19:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:19:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 10 monclient: dump: 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4786 (secure 0 0 0) 0x7fa7300123d0 con 0x7fa740134d90 2024-09-16T07:19:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:19:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740134d90 msgr2=0x7fa7401351a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740134d90 0x7fa7401351a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa730000f60 tx=0x7fa7300058a0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 1 -- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740134d90 msgr2=0x7fa7401351a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:19:04.302 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:19:04.302 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:19:04.302 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:19:04.302 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:19:04.302 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 12, 2024-09-16T07:19:04.302 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 27, 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:04.303 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:04.304 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:19:04.305 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:04.306 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.307 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.308 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:19:04.309 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.310 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:04.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:19:04.312 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.313 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:04.314 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:19:04.315 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:19:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa73cff9700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa740134d90 0x7fa7401351a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 1 -- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa724035600 msgr2=0x7fa724037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa724035600 0x7fa724037ac0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa7400a1270 tx=0x7fa7380094f0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 10 monclient: shutdown 2024-09-16T07:19:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 1 -- 172.21.15.37:0/1769459624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7400a3240 msgr2=0x7fa7401cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 1 --2- 172.21.15.37:0/1769459624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7400a3240 0x7fa7401cab70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa74006b870 tx=0x7fa728004770 comp rx=0 tx=0).stop 2024-09-16T07:19:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa744a5f700 1 -- 172.21.15.37:0/1769459624 reap_dead start 2024-09-16T07:19:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa744a5f700 1 -- 172.21.15.37:0/1769459624 reap_dead start 2024-09-16T07:19:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 1 -- 172.21.15.37:0/1769459624 shutdown_connections 2024-09-16T07:19:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 1 -- 172.21.15.37:0/1769459624 >> 172.21.15.37:0/1769459624 conn(0x7fa7400a3ea0 msgr2=0x7fa74009ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.280+0000 7fa745260700 1 -- 172.21.15.37:0/1769459624 shutdown_connections 2024-09-16T07:19:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.284+0000 7fa745260700 1 -- 172.21.15.37:0/1769459624 wait complete. 2024-09-16T07:19:04.318 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:19:04.318 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T07:19:04.319 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T07:19:04.319 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T07:19:04.319 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T07:19:04.319 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T07:19:04.319 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:19:04.369 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T07:19:04.369 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T07:19:04.369 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:19:04.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient: get_monmap_and_config 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient: build_initial_monmap 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient: monmap: 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:04.495866+0000 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:04.495866+0000 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:04.492 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:04.493 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 1 Processor -- start 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 1 -- start start 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient: init 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5bb0 0x7f53180a5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient(hunting): picked mon.noname-b con 0x7f53180a5bb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f53180a5780 0x7f5318135220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient(hunting): picked mon.noname-c con 0x7f53180a5780 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53180a64d0 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531809a5a0 con 0x7f53180a64d0 2024-09-16T07:19:04.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53180a16f0 con 0x7f53180a5780 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5318099d00 con 0x7f53180a5bb0 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient(hunting): authenticate will time out at 864.982590s 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5bb0 0x7f53180a5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5bb0 0x7f53180a5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38078/0 (socket says 172.21.15.37:38078) 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 1 -- 172.21.15.37:0/1734974475 learned_addr learned my addr 172.21.15.37:0/1734974475 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 10 monclient(hunting): get_auth_request con 0x7f53180a5bb0 auth_method 0 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f53177fe700 1 --2- 172.21.15.37:0/1734974475 >> v2:172.21.15.143:3300/0 conn(0x7f53180a5780 0x7f5318135220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f53177fe700 10 monclient(hunting): get_auth_request con 0x7f53180a5780 auth_method 0 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f53177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f53177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f53177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 10 monclient(hunting): handle_auth_done global_id 4379 payload 274 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 1 -- 172.21.15.37:0/1734974475 >> v2:172.21.15.143:3300/0 conn(0x7f53180a5780 msgr2=0x7f5318135220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 1 --2- 172.21.15.37:0/1734974475 >> v2:172.21.15.143:3300/0 conn(0x7f53180a5780 0x7f5318135220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 1 -- 172.21.15.37:0/1734974475 >> v1:172.21.15.143:6789/0 conn(0x7f53180a64d0 legacy=0x7f5318135760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 10 monclient: _finish_hunting 0 2024-09-16T07:19:04.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 1 monclient: found mon.noname-b 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 1 -- 172.21.15.37:0/1734974475 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5318099fe0 con 0x7f53180a5bb0 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5317fff700 1 --2- 172.21.15.37:0/1734974475 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5bb0 0x7f53180a5f90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5308009910 tx=0x7f5308009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0239e07242d0b7b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 1 -- 172.21.15.37:0/1734974475 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530800ba60 con 0x7f53180a5bb0 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 10 monclient: dump: 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.499 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 10 monclient: _finish_auth 0 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 5 monclient: authenticate success, global_id 4379 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 1 -- 172.21.15.37:0/1734974475 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530800bbc0 con 0x7f53180a5bb0 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:04.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f531d2c1700 4 set_mon_vals no callback set 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 1 -- 172.21.15.37:0/1734974475 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308009e60 con 0x7f53180a5bb0 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.492+0000 7f5316ffd700 10 monclient: dump: 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.501 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient: shutdown 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 -- 172.21.15.37:0/1734974475 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5bb0 msgr2=0x7f53180a5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 --2- 172.21.15.37:0/1734974475 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5bb0 0x7f53180a5f90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5308009910 tx=0x7f5308009c20 comp rx=0 tx=0).stop 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 -- 172.21.15.37:0/1734974475 shutdown_connections 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 --2- 172.21.15.37:0/1734974475 >> v2:172.21.15.143:3300/0 conn(0x7f53180a5780 0x7f5318135220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 --2- 172.21.15.37:0/1734974475 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5bb0 0x7f53180a5f90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 -- 172.21.15.37:0/1734974475 >> 172.21.15.37:0/1734974475 conn(0x7f53180a4480 msgr2=0x7f53180a4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 -- 172.21.15.37:0/1734974475 shutdown_connections 2024-09-16T07:19:04.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 -- 172.21.15.37:0/1734974475 wait complete. 2024-09-16T07:19:04.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient: build_initial_monmap 2024-09-16T07:19:04.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient: monmap: 2024-09-16T07:19:04.506 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:04.506 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:04.506 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:04.506 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:04.506 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:04.506 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:04.506 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 Processor -- start 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 -- start start 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient: init 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5780 0x7f5318134370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient(hunting): picked mon.noname-c con 0x7f53180a5780 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53180a5bb0 0x7f531811eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient(hunting): picked mon.noname-b con 0x7f53180a5bb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:04.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53180a64d0 0x7f531811f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient(hunting): picked mon.noname-a con 0x7f53180a64d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53181cae60 con 0x7f53180a64d0 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53180a4120 con 0x7f53180a5bb0 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53181227f0 con 0x7f53180a5780 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531d2c1700 10 monclient(hunting): authenticate will time out at 864.987060s 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5780 0x7f5318134370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5780 0x7f5318134370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38082/0 (socket says 172.21.15.37:38082) 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531cac0700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53180a64d0 msgr2=0x7f531811f0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:04.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 1 -- 172.21.15.37:0/2628982397 learned_addr learned my addr 172.21.15.37:0/2628982397 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 10 monclient(hunting): get_auth_request con 0x7f53180a5780 auth_method 0 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f531cac0700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53180a64d0 0x7f531811f0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f53177fe700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53180a5bb0 0x7f531811eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f53177fe700 10 monclient(hunting): get_auth_request con 0x7f53180a5bb0 auth_method 0 2024-09-16T07:19:04.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f53177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:04.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f53177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:04.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f53177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:04.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2024-09-16T07:19:04.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 1 -- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53180a5bb0 msgr2=0x7f531811eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53180a5bb0 0x7f531811eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 1 -- 172.21.15.37:0/2628982397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53180a64d0 msgr2=0x7f531811f0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:19:04.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53180a64d0 0x7f531811f0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 10 monclient: _finish_hunting 0 2024-09-16T07:19:04.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 1 monclient: found mon.noname-c 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f5317fff700 1 -- 172.21.15.37:0/2628982397 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5308009570 con 0x7f53180a5780 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.496+0000 7f53177fe700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53180a5bb0 0x7f531811eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5317fff700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5780 0x7f5318134370 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f530800b3c0 tx=0x7f5308004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 1 -- 172.21.15.37:0/2628982397 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308004400 con 0x7f53180a5780 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 10 monclient: dump: 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.511 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 10 monclient: _finish_auth 0 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 1 -- 172.21.15.37:0/2628982397 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530801f070 con 0x7f53180a5780 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 5 monclient: authenticate success, global_id 4382 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 10 monclient: _renew_subs 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:04.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 1 -- 172.21.15.37:0/2628982397 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53180a0520 con 0x7f53180a5780 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 1 -- 172.21.15.37:0/2628982397 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308003900 con 0x7f53180a5780 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 10 monclient: dump: 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:04.513 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 10 monclient: _renew_subs 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f53167fc700 4 set_mon_vals no callback set 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 1 -- 172.21.15.37:0/2628982397 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5318121050 con 0x7f53180a5780 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 1 -- 172.21.15.37:0/2628982397 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f5308021030 con 0x7f53180a5780 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f531d2c1700 1 -- 172.21.15.37:0/2628982397 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53181be360 con 0x7f53180a5780 2024-09-16T07:19:04.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f53040355b0 0x7f5304037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:04.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f5314ff9700 1 -- 172.21.15.37:0/2628982397 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f5308048e30 con 0x7f53180a5780 2024-09-16T07:19:04.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f53177fe700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f53040355b0 0x7f5304037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:04.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f53177fe700 10 monclient: get_auth_request con 0x7f53040355b0 auth_method 0 2024-09-16T07:19:04.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.500+0000 7f53177fe700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f53040355b0 0x7f5304037a70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5318122a00 tx=0x7f530c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:04.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.508+0000 7f5314ff9700 1 -- 172.21.15.37:0/2628982397 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f53181be360 con 0x7f53180a5780 2024-09-16T07:19:04.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.508+0000 7f5314ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:04.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.508+0000 7f5314ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:04.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:19:04.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:19:04.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:04.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 -- 172.21.15.37:0/2628982397 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f53181be360 con 0x7f53180a5780 2024-09-16T07:19:04.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f5314ff9700 1 -- 172.21.15.37:0/2628982397 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f53181be360 con 0x7f53180a5780 2024-09-16T07:19:04.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f5314ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:19:04.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f5314ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:04.741 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 -- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f53040355b0 msgr2=0x7f5304037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f53040355b0 0x7f5304037a70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5318122a00 tx=0x7f530c008040 comp rx=0 tx=0).stop 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 10 monclient: shutdown 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 -- 172.21.15.37:0/2628982397 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5780 msgr2=0x7f5318134370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5780 0x7f5318134370 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f530800b3c0 tx=0x7f5308004770 comp rx=0 tx=0).stop 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 -- 172.21.15.37:0/2628982397 shutdown_connections 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53180a64d0 0x7f531811f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53180a5bb0 0x7f531811eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f53040355b0 0x7f5304037a70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 --2- 172.21.15.37:0/2628982397 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53180a5780 0x7f5318134370 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 -- 172.21.15.37:0/2628982397 >> 172.21.15.37:0/2628982397 conn(0x7f53180a4480 msgr2=0x7f53181bd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:04.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 -- 172.21.15.37:0/2628982397 shutdown_connections 2024-09-16T07:19:04.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:04.740+0000 7f531d2c1700 1 -- 172.21.15.37:0/2628982397 wait complete. 2024-09-16T07:19:07.756 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:19:07.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient: get_monmap_and_config 2024-09-16T07:19:07.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient: build_initial_monmap 2024-09-16T07:19:07.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient: monmap: 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:07.881723+0000 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:07.881723+0000 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:07.878 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:07.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 1 Processor -- start 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 1 -- start start 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient: init 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6e000a2750 0x7f6e000a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient(hunting): picked mon.noname-a con 0x7f6e000a2750 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e000a3070 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e00133730 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e00099be0 con 0x7f6e00133730 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0009de40 con 0x7f6e000a3070 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:07.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0009d530 con 0x7f6e000a2750 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e05fbc700 10 monclient(hunting): authenticate will time out at 868.367577s 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e057bb700 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f6e00133730 legacy=0x7f6e0013b310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e04fba700 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f6e000a2750 msgr2=0x7f6e000a2b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.876+0000 7f6e04fba700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6e000a2750 0x7f6e000a2b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dfffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f6e000a3070 0x7f6e00133020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.37:34698/0 (socket says 172.21.15.37:34698) 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dfffff700 1 -- 172.21.15.37:0/3238562666 learned_addr learned my addr 172.21.15.37:0/3238562666 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238315838 0 0) 0x7f6e0009de40 con 0x7f6e000a3070 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): _negotiate 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): my global_id is 4450 2024-09-16T07:19:07.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de80035d0 con 0x7f6e000a3070 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069940986 0 0) 0x7f6de80035d0 con 0x7f6e000a3070 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 >> v2:172.21.15.37:3300/0 conn(0x7f6e000a2750 msgr2=0x7f6e000a2b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 --2- 172.21.15.37:0/3238562666 >> v2:172.21.15.37:3300/0 conn(0x7f6e000a2750 0x7f6e000a2b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 >> v1:172.21.15.37:6789/0 conn(0x7f6e00133730 legacy=0x7f6e0013b310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 monclient: found mon.noname-c-legacy 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0009a020 con 0x7f6e000a3070 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: _finish_auth 0 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:07.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e05fbc700 5 monclient: authenticate success, global_id 4450 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e05fbc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 120013200 0 0) 0x7f6df0001670 con 0x7f6e000a3070 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: dump: 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:07.893 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 >> v1:172.21.15.143:6789/0 conn(0x7f6e000a3070 legacy=0x7f6e00133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6de8003a20 0x7f6de8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): picked mon.c con 0x7f6de8003a20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8004870 0x7f6de8004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): picked mon.a con 0x7f6de8004870 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6de80053e0 0x7f6de800ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dfffff700 1 -- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8004870 msgr2=0x7f6de8004cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): picked mon.b con 0x7f6de80053e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dfffff700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8004870 0x7f6de8004cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0009d530 con 0x7f6de8004870 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de800c330 con 0x7f6de80053e0 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de800c4f0 con 0x7f6de8003a20 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient(hunting): _finish_auth 0 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e05fbc700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6de8003a20 0x7f6de8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient(hunting): get_auth_request con 0x7f6de8003a20 auth_method 0 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e057bb700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6de80053e0 0x7f6de800ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e057bb700 10 monclient(hunting): get_auth_request con 0x7f6de80053e0 auth_method 0 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e057bb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e057bb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e057bb700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e057bb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e057bb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e057bb700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient(hunting): handle_auth_done global_id 4450 payload 293 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 1 -- 172.21.15.37:0/3238562666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6de80053e0 msgr2=0x7f6de800ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6de80053e0 0x7f6de800ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 1 -- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8004870 msgr2=0x7f6de8004cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8004870 0x7f6de8004cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e057bb700 1 -- 172.21.15.37:0/3238562666 reap_dead start 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient: _finish_hunting 0 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 1 monclient: found mon.c 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 1 -- 172.21.15.37:0/3238562666 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de80035d0 con 0x7f6de8003a20 2024-09-16T07:19:07.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient: _finish_auth 0 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 10 monclient: _check_auth_tickets 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e04fba700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6de8003a20 0x7f6de8003e20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009e30 tx=0x7f6de80058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f47af27595a8bb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df40098d0 con 0x7f6de8003a20 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: dump: 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:07.898 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e05fbc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df400bd60 con 0x7f6de8003a20 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e05fbc700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 1 -- 172.21.15.37:0/3238562666 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df4003ca0 con 0x7f6de8003a20 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6e05fbc700 4 set_mon_vals no callback set 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:07.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.880+0000 7f6dff7fe700 10 monclient: dump: 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient: shutdown 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:07.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6de8003a20 msgr2=0x7f6de8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6de8003a20 0x7f6de8003e20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009e30 tx=0x7f6de80058a0 comp rx=0 tx=0).stop 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3238562666 shutdown_connections 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 --2- 172.21.15.37:0/3238562666 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6de8003a20 0x7f6de8003e20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3238562666 >> 172.21.15.37:0/3238562666 conn(0x7f6e000a3ea0 msgr2=0x7f6e000a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3238562666 shutdown_connections 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3238562666 wait complete. 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient: build_initial_monmap 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient: monmap: 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:07.901 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 Processor -- start 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 -- start start 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient: init 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:19:07.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8003a20 0x7f6e001c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient(hunting): picked mon.noname-a con 0x7f6de8003a20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e00132500 0x7f6e00132960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient(hunting): picked mon.noname-b con 0x7f6e00132500 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e0011bfb0 0x7f6e0011c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient(hunting): picked mon.noname-c con 0x7f6e0011bfb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0011f9b0 con 0x7f6de8003a20 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e00132ea0 con 0x7f6e00132500 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0011fc00 con 0x7f6e0011bfb0 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:07.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e05fbc700 10 monclient(hunting): authenticate will time out at 868.375397s 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e04fba700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8003a20 msgr2=0x7f6e001c7fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e04fba700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8003a20 0x7f6e001c7fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e0011bfb0 0x7f6e0011c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e0011bfb0 0x7f6e0011c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38092/0 (socket says 172.21.15.37:38092) 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 1 -- 172.21.15.37:0/3488526878 learned_addr learned my addr 172.21.15.37:0/3488526878 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 10 monclient(hunting): get_auth_request con 0x7f6e0011bfb0 auth_method 0 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6dfffff700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e00132500 0x7f6e00132960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:07.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6dfffff700 10 monclient(hunting): get_auth_request con 0x7f6e00132500 auth_method 0 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6dfffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6dfffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6dfffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.884+0000 7f6e057bb700 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e057bb700 1 -- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e00132500 msgr2=0x7f6e00132960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e057bb700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e00132500 0x7f6e00132960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e057bb700 1 -- 172.21.15.37:0/3488526878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8003a20 msgr2=0x7f6e001c7fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e057bb700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8003a20 0x7f6e001c7fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e057bb700 10 monclient: _finish_hunting 0 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e057bb700 1 monclient: found mon.noname-c 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e057bb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:07.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e057bb700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e057bb700 1 -- 172.21.15.37:0/3488526878 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e001330b0 con 0x7f6e0011bfb0 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfffff700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e00132500 0x7f6e00132960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e057bb700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e0011bfb0 0x7f6e0011c410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6de8005850 tx=0x7f6df800ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 1 -- 172.21.15.37:0/3488526878 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df800cbd0 con 0x7f6e0011bfb0 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 10 monclient: dump: 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:07.906 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 10 monclient: _finish_auth 0 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 10 monclient: _check_auth_tickets 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 1 -- 172.21.15.37:0/3488526878 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df800cd30 con 0x7f6e0011bfb0 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 5 monclient: authenticate success, global_id 4385 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 10 monclient: _renew_subs 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3488526878 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0006cf90 con 0x7f6e0011bfb0 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:07.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 1 -- 172.21.15.37:0/3488526878 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8017660 con 0x7f6e0011bfb0 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 10 monclient: dump: 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:07.908 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfeffd700 4 set_mon_vals no callback set 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 10 monclient: _renew_subs 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3488526878 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0011fee0 con 0x7f6e0011bfb0 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 1 -- 172.21.15.37:0/3488526878 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f6df800b3a0 con 0x7f6e0011bfb0 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3488526878 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0011fd40 con 0x7f6e0011bfb0 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6dec0355b0 0x7f6dec037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6dfd7fa700 1 -- 172.21.15.37:0/3488526878 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f6df8048210 con 0x7f6e0011bfb0 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e04fba700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6dec0355b0 0x7f6dec037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:07.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e04fba700 10 monclient: get_auth_request con 0x7f6dec0355b0 auth_method 0 2024-09-16T07:19:07.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.888+0000 7f6e04fba700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6dec0355b0 0x7f6dec037a70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009e30 tx=0x7f6df4019040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:07.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.900+0000 7f6dfd7fa700 1 -- 172.21.15.37:0/3488526878 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6e0011fd40 con 0x7f6e0011bfb0 2024-09-16T07:19:07.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.900+0000 7f6dfd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:07.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:07.900+0000 7f6dfd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:08.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:19:08.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:19:08.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:08.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3488526878 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6e0011fd40 con 0x7f6e0011bfb0 2024-09-16T07:19:08.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6dfd7fa700 1 -- 172.21.15.37:0/3488526878 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f6e0011fd40 con 0x7f6e0011bfb0 2024-09-16T07:19:08.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6dfd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:19:08.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6dfd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:08.133 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:19:08.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6dec0355b0 msgr2=0x7f6dec037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:08.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6dec0355b0 0x7f6dec037a70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009e30 tx=0x7f6df4019040 comp rx=0 tx=0).stop 2024-09-16T07:19:08.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 10 monclient: shutdown 2024-09-16T07:19:08.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:08.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3488526878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e0011bfb0 msgr2=0x7f6e0011c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:08.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e0011bfb0 0x7f6e0011c410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6de8005850 tx=0x7f6df800ecc0 comp rx=0 tx=0).stop 2024-09-16T07:19:08.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3488526878 shutdown_connections 2024-09-16T07:19:08.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6dec0355b0 0x7f6dec037a70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:08.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e0011bfb0 0x7f6e0011c410 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:08.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e00132500 0x7f6e00132960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:08.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 --2- 172.21.15.37:0/3488526878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6de8003a20 0x7f6e001c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:08.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3488526878 >> 172.21.15.37:0/3488526878 conn(0x7f6e000a3ea0 msgr2=0x7f6e0009f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:08.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3488526878 shutdown_connections 2024-09-16T07:19:08.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:08.132+0000 7f6e05fbc700 1 -- 172.21.15.37:0/3488526878 wait complete. 2024-09-16T07:19:11.152 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:19:11.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient: get_monmap_and_config 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient: build_initial_monmap 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient: monmap: 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:11.275949+0000 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:11.275949+0000 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:11.272 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:11.273 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:11.273 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:11.273 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:11.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 1 Processor -- start 2024-09-16T07:19:11.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 1 -- start start 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient: init 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a2750 0x7fea100999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient(hunting): picked mon.noname-b con 0x7fea100a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fea10071880 0x7fea10099f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient(hunting): picked mon.noname-a con 0x7fea10071880 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea100a31e0 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea100a2b30 con 0x7fea100a31e0 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1009d0e0 con 0x7fea10071880 2024-09-16T07:19:11.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1009bb40 con 0x7fea100a2750 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient(hunting): authenticate will time out at 871.761798s 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0effd700 1 -- >> v2:172.21.15.37:3300/0 conn(0x7fea10071880 msgr2=0x7fea10099f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0ffff700 1 -- >> v1:172.21.15.37:6789/0 conn(0x7fea100a31e0 legacy=0x7fea10138040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0effd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fea10071880 0x7fea10099f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a2750 0x7fea100999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a2750 0x7fea100999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41968/0 (socket says 172.21.15.37:41968) 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 1 -- 172.21.15.37:0/2572514755 learned_addr learned my addr 172.21.15.37:0/2572514755 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 10 monclient(hunting): get_auth_request con 0x7fea100a2750 auth_method 0 2024-09-16T07:19:11.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 10 monclient(hunting): handle_auth_done global_id 4388 payload 274 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 1 -- 172.21.15.37:0/2572514755 >> v2:172.21.15.37:3300/0 conn(0x7fea10071880 msgr2=0x7fea10099f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 1 --2- 172.21.15.37:0/2572514755 >> v2:172.21.15.37:3300/0 conn(0x7fea10071880 0x7fea10099f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 1 -- 172.21.15.37:0/2572514755 >> v1:172.21.15.37:6789/0 conn(0x7fea100a31e0 legacy=0x7fea10138040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 1 monclient: found mon.noname-b 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:11.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 1 -- 172.21.15.37:0/2572514755 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1009c3e0 con 0x7fea100a2750 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0f7fe700 1 --2- 172.21.15.37:0/2572514755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a2750 0x7fea100999e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fea1009aaa0 tx=0x7fea00009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98ea9af5c11f1582 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 1 -- 172.21.15.37:0/2572514755 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0000b860 con 0x7fea100a2750 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 10 monclient: dump: 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:11.280 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 10 monclient: _finish_auth 0 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 5 monclient: authenticate success, global_id 4388 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 1 -- 172.21.15.37:0/2572514755 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0000b9c0 con 0x7fea100a2750 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea14cb6700 4 set_mon_vals no callback set 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 1 -- 172.21.15.37:0/2572514755 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0000bd50 con 0x7fea100a2750 2024-09-16T07:19:11.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.272+0000 7fea0e7fc700 10 monclient: dump: 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:11.282 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:11.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient: shutdown 2024-09-16T07:19:11.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:11.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 -- 172.21.15.37:0/2572514755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a2750 msgr2=0x7fea100999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:11.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 --2- 172.21.15.37:0/2572514755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a2750 0x7fea100999e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fea1009aaa0 tx=0x7fea00009bf0 comp rx=0 tx=0).stop 2024-09-16T07:19:11.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 -- 172.21.15.37:0/2572514755 shutdown_connections 2024-09-16T07:19:11.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 --2- 172.21.15.37:0/2572514755 >> v2:172.21.15.37:3300/0 conn(0x7fea10071880 0x7fea10099f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:11.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 --2- 172.21.15.37:0/2572514755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a2750 0x7fea100999e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:11.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 -- 172.21.15.37:0/2572514755 >> 172.21.15.37:0/2572514755 conn(0x7fea100a3ea0 msgr2=0x7fea100a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:11.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 -- 172.21.15.37:0/2572514755 shutdown_connections 2024-09-16T07:19:11.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 -- 172.21.15.37:0/2572514755 wait complete. 2024-09-16T07:19:11.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient: build_initial_monmap 2024-09-16T07:19:11.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient: monmap: 2024-09-16T07:19:11.284 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:11.284 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 Processor -- start 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 -- start start 2024-09-16T07:19:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient: init 2024-09-16T07:19:11.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:11.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:19:11.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea10071880 0x7fea1011eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient(hunting): picked mon.noname-b con 0x7fea10071880 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea100a2750 0x7fea1011f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient(hunting): picked mon.noname-a con 0x7fea100a2750 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a31e0 0x7fea1011f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient(hunting): picked mon.noname-c con 0x7fea100a31e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea100724c0 con 0x7fea100a2750 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10121060 con 0x7fea10071880 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1009f8b0 con 0x7fea100a31e0 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:11.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea14cb6700 10 monclient(hunting): authenticate will time out at 871.766669s 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0effd700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea100a2750 msgr2=0x7fea1011f090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea100a2750 0x7fea1011f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a31e0 0x7fea1011f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a31e0 0x7fea1011f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41972/0 (socket says 172.21.15.37:41972) 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 1 -- 172.21.15.37:0/458631244 learned_addr learned my addr 172.21.15.37:0/458631244 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 10 monclient(hunting): get_auth_request con 0x7fea100a31e0 auth_method 0 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0f7fe700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea10071880 0x7fea1011eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:11.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0f7fe700 10 monclient(hunting): get_auth_request con 0x7fea10071880 auth_method 0 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 1 -- 172.21.15.37:0/458631244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea10071880 msgr2=0x7fea1011eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea10071880 0x7fea1011eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 1 -- 172.21.15.37:0/458631244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea100a2750 msgr2=0x7fea1011f090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea100a2750 0x7fea1011f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 10 monclient: _finish_hunting 0 2024-09-16T07:19:11.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 1 monclient: found mon.noname-c 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.276+0000 7fea0ffff700 1 -- 172.21.15.37:0/458631244 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea00009570 con 0x7fea100a31e0 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea0ffff700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a31e0 0x7fea1011f5d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fea0800b480 tx=0x7fea0800b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 1 -- 172.21.15.37:0/458631244 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08003e70 con 0x7fea100a31e0 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 10 monclient: dump: 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:11.293 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 10 monclient: _finish_auth 0 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 10 monclient: _check_auth_tickets 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 1 -- 172.21.15.37:0/458631244 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea080060d0 con 0x7fea100a31e0 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 5 monclient: authenticate success, global_id 4391 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 1 -- 172.21.15.37:0/458631244 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08006480 con 0x7fea100a31e0 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 10 monclient: _renew_subs 2024-09-16T07:19:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 1 -- 172.21.15.37:0/458631244 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea10071030 con 0x7fea100a31e0 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea0dffb700 4 set_mon_vals no callback set 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 10 monclient: dump: 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:11.296 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 10 monclient: _renew_subs 2024-09-16T07:19:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 1 -- 172.21.15.37:0/458631244 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea10122a30 con 0x7fea100a31e0 2024-09-16T07:19:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea14cb6700 1 -- 172.21.15.37:0/458631244 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea10122190 con 0x7fea100a31e0 2024-09-16T07:19:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 1 -- 172.21.15.37:0/458631244 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fea080065e0 con 0x7fea100a31e0 2024-09-16T07:19:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe9f00355b0 0x7fe9f0037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:11.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea07fff700 1 -- 172.21.15.37:0/458631244 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4738+0+0 (secure 0 0 0) 0x7fea08046b60 con 0x7fea100a31e0 2024-09-16T07:19:11.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea0f7fe700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe9f00355b0 0x7fe9f0037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:11.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea0f7fe700 10 monclient: get_auth_request con 0x7fe9f00355b0 auth_method 0 2024-09-16T07:19:11.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.280+0000 7fea0f7fe700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe9f00355b0 0x7fe9f0037a70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea00009150 tx=0x7fea00008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:11.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.292+0000 7fea07fff700 1 -- 172.21.15.37:0/458631244 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fea08018cd0 con 0x7fea100a31e0 2024-09-16T07:19:11.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.292+0000 7fea07fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:11.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.292+0000 7fea07fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:11.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.528+0000 7fea14cb6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:19:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.528+0000 7fea14cb6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:19:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.528+0000 7fea14cb6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.528+0000 7fea14cb6700 1 -- 172.21.15.37:0/458631244 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fea100a0a50 con 0x7fea100a31e0 2024-09-16T07:19:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.528+0000 7fea07fff700 1 -- 172.21.15.37:0/458631244 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fea08017030 con 0x7fea100a31e0 2024-09-16T07:19:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.528+0000 7fea07fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:19:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.528+0000 7fea07fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:11.530 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":34,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:19:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 -- 172.21.15.37:0/458631244 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe9f00355b0 msgr2=0x7fe9f0037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe9f00355b0 0x7fe9f0037a70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea00009150 tx=0x7fea00008030 comp rx=0 tx=0).stop 2024-09-16T07:19:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 10 monclient: shutdown 2024-09-16T07:19:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 -- 172.21.15.37:0/458631244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a31e0 msgr2=0x7fea1011f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a31e0 0x7fea1011f5d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fea0800b480 tx=0x7fea0800b790 comp rx=0 tx=0).stop 2024-09-16T07:19:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 -- 172.21.15.37:0/458631244 shutdown_connections 2024-09-16T07:19:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe9f00355b0 0x7fe9f0037a70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea100a31e0 0x7fea1011f5d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:11.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea100a2750 0x7fea1011f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 --2- 172.21.15.37:0/458631244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea10071880 0x7fea1011eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 -- 172.21.15.37:0/458631244 >> 172.21.15.37:0/458631244 conn(0x7fea100a3ea0 msgr2=0x7fea1006f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 -- 172.21.15.37:0/458631244 shutdown_connections 2024-09-16T07:19:11.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:11.532+0000 7fea14cb6700 1 -- 172.21.15.37:0/458631244 wait complete. 2024-09-16T07:19:14.545 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:19:14.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient: get_monmap_and_config 2024-09-16T07:19:14.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient: build_initial_monmap 2024-09-16T07:19:14.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient: monmap: 2024-09-16T07:19:14.668 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:14.668 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:14.668 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:14.671892+0000 2024-09-16T07:19:14.668 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:14.671892+0000 2024-09-16T07:19:14.668 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:14.668 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:14.668 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:14.668 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:14.669 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:14.669 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:14.669 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:14.669 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 1 Processor -- start 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 1 -- start start 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient: init 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient(hunting): picked mon.noname-c con 0x7f3f040a0a50 addr v2:172.21.15.143:3300/0 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f040a2c90 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f040a30c0 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0409c570 con 0x7f3f040a30c0 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0409b890 con 0x7f3f040a2c90 2024-09-16T07:19:14.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0413a630 con 0x7f3f040a0a50 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0b014700 10 monclient(hunting): authenticate will time out at 875.157883s 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0a813700 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f3f040a30c0 legacy=0x7f3f04139f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0a012700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f09811700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f3f040a2c90 0x7f3f04139810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.37:47368/0 (socket says 172.21.15.37:47368) 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0a012700 10 monclient(hunting): get_auth_request con 0x7f3f040a0a50 auth_method 0 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f09811700 1 -- 172.21.15.37:0/845695313 learned_addr learned my addr 172.21.15.37:0/845695313 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0a012700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0a012700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0a012700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 msgr2=0x7f3f040a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:19:14.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 msgr2=0x7f3f040a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:19:14.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:19:14.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.668+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:14.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.872+0000 7f3f0a813700 1 -- 172.21.15.37:0/845695313 >> v1:172.21.15.37:6789/0 conn(0x7f3f040a30c0 legacy=0x7f3f04139f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:19:14.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.872+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:14.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.872+0000 7f3f0a012700 10 monclient(hunting): get_auth_request con 0x7f3f040a0a50 auth_method 0 2024-09-16T07:19:14.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.872+0000 7f3f0a012700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:14.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.872+0000 7f3f0a012700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:14.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.872+0000 7f3f0a012700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:19:14.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.872+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 msgr2=0x7f3f040a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:19:14.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.872+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 msgr2=0x7f3f040a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:19:14.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.872+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:19:14.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:14.872+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:19:15.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:15.276+0000 7f3f0a813700 1 -- 172.21.15.37:0/845695313 >> v1:172.21.15.37:6789/0 conn(0x7f3f040a30c0 legacy=0x7f3f04139f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:19:15.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:15.276+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:15.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:15.276+0000 7f3f0a012700 10 monclient(hunting): get_auth_request con 0x7f3f040a0a50 auth_method 0 2024-09-16T07:19:15.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:15.276+0000 7f3f0a012700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:15.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:15.276+0000 7f3f0a012700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:15.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:15.276+0000 7f3f0a012700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:19:15.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:15.276+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 msgr2=0x7f3f040a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:19:15.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:15.276+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 msgr2=0x7f3f040a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:19:15.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:15.276+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:19:15.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:15.276+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:19:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:16.080+0000 7f3f0a813700 1 -- 172.21.15.37:0/845695313 >> v1:172.21.15.37:6789/0 conn(0x7f3f040a30c0 legacy=0x7f3f04139f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:19:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:16.080+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:16.080+0000 7f3f0a012700 10 monclient(hunting): get_auth_request con 0x7f3f040a0a50 auth_method 0 2024-09-16T07:19:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:16.080+0000 7f3f0a012700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:16.080+0000 7f3f0a012700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:16.080+0000 7f3f0a012700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:19:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:16.080+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 msgr2=0x7f3f040a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:19:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:16.080+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 msgr2=0x7f3f040a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:19:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:16.080+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:19:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:16.080+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T07:19:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 10 monclient(hunting): tick 2024-09-16T07:19:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 10 monclient(hunting): _check_auth_tickets 2024-09-16T07:19:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 1 monclient(hunting): continuing hunt 2024-09-16T07:19:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 msgr2=0x7f3f040a2750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.143:3300/0 conn(0x7f3f040a0a50 0x7f3f040a2750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 1 -- 172.21.15.37:0/845695313 >> v1:172.21.15.143:6789/0 conn(0x7f3f040a2c90 legacy=0x7f3f04139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 1 -- 172.21.15.37:0/845695313 >> v1:172.21.15.37:6789/0 conn(0x7f3f040a30c0 legacy=0x7f3f04139f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 10 monclient: _add_conns ranks=[2,3] 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f3ee8003720 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 0x7f3ee8008760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f3ee80062a0 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 1 -- 172.21.15.37:0/845695313 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0413a630 con 0x7f3ee80062a0 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 1 -- 172.21.15.37:0/845695313 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8008e20 con 0x7f3ee8003720 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3efbfff700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f09811700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 msgr2=0x7f3ee8008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f09811700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 0x7f3ee8008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f0a012700 10 monclient(hunting): get_auth_request con 0x7f3ee8003720 auth_method 0 2024-09-16T07:19:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f0a012700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f0a012700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f0a012700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 msgr2=0x7f3ee8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:19:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 msgr2=0x7f3ee8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:19:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:19:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.668+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:17.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f09811700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 msgr2=0x7f3ee8008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:17.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f09811700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 0x7f3ee8008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:19:17.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:17.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f0a012700 10 monclient(hunting): get_auth_request con 0x7f3ee8003720 auth_method 0 2024-09-16T07:19:17.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f0a012700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:17.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f0a012700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:17.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f0a012700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:19:17.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 msgr2=0x7f3ee8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:19:17.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 msgr2=0x7f3ee8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:19:17.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:19:17.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:17.872+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:19:18.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f09811700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 msgr2=0x7f3ee8008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:18.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f09811700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 0x7f3ee8008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:19:18.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:18.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f0a012700 10 monclient(hunting): get_auth_request con 0x7f3ee8003720 auth_method 0 2024-09-16T07:19:18.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f0a012700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:18.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f0a012700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:18.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f0a012700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:19:18.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 msgr2=0x7f3ee8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T07:19:18.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 msgr2=0x7f3ee8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:19:18.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:19:18.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:18.276+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:19:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24227]: starting. 2024-09-16T07:19:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24227]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24226 2024-09-16T07:19:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24227]: created object 0... 2024-09-16T07:19:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24227]: created object 25... 2024-09-16T07:19:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24227]: created object 49... 2024-09-16T07:19:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24227]: finishing. 2024-09-16T07:19:18.832 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24227]: shutting down. 2024-09-16T07:19:19.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09811700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 msgr2=0x7f3ee8008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:19.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09811700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 0x7f3ee8008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T07:19:19.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 10 monclient(hunting): get_auth_request con 0x7f3ee8003720 auth_method 0 2024-09-16T07:19:19.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:19:19.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:19.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 10 monclient(hunting): handle_auth_done global_id 4394 payload 274 2024-09-16T07:19:19.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 msgr2=0x7f3ee8008760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:19.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> v2:172.21.15.37:3300/0 conn(0x7f3ee80062a0 0x7f3ee8008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 10 monclient: _finish_hunting 0 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 1 monclient: found mon.noname-b 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 1 -- 172.21.15.37:0/845695313 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f04099be0 con 0x7f3ee8003720 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a012700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009100 tx=0x7f3ef4018d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=560689ce36699e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 1 -- 172.21.15.37:0/845695313 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef401d070 con 0x7f3ee8003720 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 10 monclient: dump: 2024-09-16T07:19:19.083 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 10 monclient: _finish_auth 0 2024-09-16T07:19:19.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 10 monclient: _check_auth_tickets 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 5 monclient: authenticate success, global_id 4394 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 1 -- 172.21.15.37:0/845695313 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef400c040 con 0x7f3ee8003720 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 4 set_mon_vals no callback set 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 1 -- 172.21.15.37:0/845695313 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4004270 con 0x7f3ee8003720 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:19.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f09010700 10 monclient: dump: 2024-09-16T07:19:19.086 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.086 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.086 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.086 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.086 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.086 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.086 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.086 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.086 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.086 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 10 monclient: shutdown 2024-09-16T07:19:19.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:19.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 1 -- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 msgr2=0x7f3ee8005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 1 --2- 172.21.15.37:0/845695313 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ee8003720 0x7f3ee8005b70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4009100 tx=0x7f3ef4018d70 comp rx=0 tx=0).stop 2024-09-16T07:19:19.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0a813700 1 -- 172.21.15.37:0/845695313 reap_dead start 2024-09-16T07:19:19.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 1 -- 172.21.15.37:0/845695313 shutdown_connections 2024-09-16T07:19:19.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 1 -- 172.21.15.37:0/845695313 >> 172.21.15.37:0/845695313 conn(0x7f3f040a3ea0 msgr2=0x7f3f040a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:19.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 1 -- 172.21.15.37:0/845695313 shutdown_connections 2024-09-16T07:19:19.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.080+0000 7f3f0b014700 1 -- 172.21.15.37:0/845695313 wait complete. 2024-09-16T07:19:19.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient: build_initial_monmap 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient: monmap: 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:19.092 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 1 Processor -- start 2024-09-16T07:19:19.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 1 -- start start 2024-09-16T07:19:19.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient: init 2024-09-16T07:19:19.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:19.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:19:19.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f04138c80 0x7f3f041ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient(hunting): picked mon.noname-c con 0x7f3f04138c80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:19.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f04139060 0x7f3f04122920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient(hunting): picked mon.noname-a con 0x7f3f04139060 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:19.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f04139480 0x7f3f04122e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient(hunting): picked mon.noname-b con 0x7f3f04139480 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f041263f0 con 0x7f3f04139060 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f04126640 con 0x7f3f04139480 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0409fcc0 con 0x7f3f04138c80 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0b014700 10 monclient(hunting): authenticate will time out at 879.575536s 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f09811700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f04139060 msgr2=0x7f3f04122920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:19.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f09811700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f04139060 0x7f3f04122920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0a012700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f04138c80 0x7f3f041ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0a012700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f04138c80 0x7f3f041ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:42022/0 (socket says 172.21.15.37:42022) 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0a012700 1 -- 172.21.15.37:0/4036667886 learned_addr learned my addr 172.21.15.37:0/4036667886 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0a012700 10 monclient(hunting): get_auth_request con 0x7f3f04138c80 auth_method 0 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0a012700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0a012700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.084+0000 7f3f0a012700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a813700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f04139480 0x7f3f04122e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a813700 10 monclient(hunting): get_auth_request con 0x7f3f04139480 auth_method 0 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a813700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a813700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a813700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a813700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a813700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a813700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 1 -- 172.21.15.37:0/4036667886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f04139480 msgr2=0x7f3f04122e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f04139480 0x7f3f04122e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 1 -- 172.21.15.37:0/4036667886 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f04139060 msgr2=0x7f3f04122920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:19:19.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f04139060 0x7f3f04122920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 10 monclient: _finish_hunting 0 2024-09-16T07:19:19.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 1 monclient: found mon.noname-c 2024-09-16T07:19:19.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:19.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 1 -- 172.21.15.37:0/4036667886 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef400e040 con 0x7f3f04138c80 2024-09-16T07:19:19.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0a012700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f04138c80 0x7f3f041ce2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4018010 tx=0x7f3ef40047e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 1 -- 172.21.15.37:0/4036667886 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4004260 con 0x7f3f04138c80 2024-09-16T07:19:19.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 10 monclient: dump: 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.098 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 10 monclient: _finish_auth 0 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 10 monclient: _check_auth_tickets 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 5 monclient: authenticate success, global_id 4397 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 10 monclient: _renew_subs 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 1 -- 172.21.15.37:0/4036667886 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0409f790 con 0x7f3f04138c80 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 1 -- 172.21.15.37:0/4036667886 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef40043c0 con 0x7f3f04138c80 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 1 -- 172.21.15.37:0/4036667886 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef40036c0 con 0x7f3f04138c80 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efbfff700 4 set_mon_vals no callback set 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 10 monclient: dump: 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.100 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.101 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 10 monclient: _renew_subs 2024-09-16T07:19:19.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 1 -- 172.21.15.37:0/4036667886 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f04072510 con 0x7f3f04138c80 2024-09-16T07:19:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 1 -- 172.21.15.37:0/4036667886 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f3ef4003820 con 0x7f3f04138c80 2024-09-16T07:19:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3ee8036360 0x7f3ee8038820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3efaffd700 1 -- 172.21.15.37:0/4036667886 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f3ef404a670 con 0x7f3f04138c80 2024-09-16T07:19:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f0b014700 1 -- 172.21.15.37:0/4036667886 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0409a2a0 con 0x7f3f04138c80 2024-09-16T07:19:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f09811700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3ee8036360 0x7f3ee8038820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.088+0000 7f3f09811700 10 monclient: get_auth_request con 0x7f3ee8036360 auth_method 0 2024-09-16T07:19:19.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.092+0000 7f3f09811700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3ee8036360 0x7f3ee8038820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3f04071040 tx=0x7f3f000017e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.096+0000 7f3efaffd700 1 -- 172.21.15.37:0/4036667886 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3ef404a070 con 0x7f3f04138c80 2024-09-16T07:19:19.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.096+0000 7f3efaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:19.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.096+0000 7f3efaffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:19.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:19:19.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:19:19.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 -- 172.21.15.37:0/4036667886 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3f04126780 con 0x7f3f04138c80 2024-09-16T07:19:19.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3efaffd700 1 -- 172.21.15.37:0/4036667886 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f3ef4043620 con 0x7f3f04138c80 2024-09-16T07:19:19.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3efaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:19:19.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3efaffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:19.333 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":14,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:19:19.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 -- 172.21.15.37:0/4036667886 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3ee8036360 msgr2=0x7f3ee8038820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3ee8036360 0x7f3ee8038820 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3f04071040 tx=0x7f3f000017e0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 10 monclient: shutdown 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 -- 172.21.15.37:0/4036667886 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f04138c80 msgr2=0x7f3f041ce2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f04138c80 0x7f3f041ce2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4018010 tx=0x7f3ef40047e0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 -- 172.21.15.37:0/4036667886 shutdown_connections 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f04139480 0x7f3f04122e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f04139060 0x7f3f04122920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3ee8036360 0x7f3ee8038820 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 --2- 172.21.15.37:0/4036667886 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f04138c80 0x7f3f041ce2f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 -- 172.21.15.37:0/4036667886 >> 172.21.15.37:0/4036667886 conn(0x7f3f040a3ea0 msgr2=0x7f3f0409e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:19.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 -- 172.21.15.37:0/4036667886 shutdown_connections 2024-09-16T07:19:19.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.332+0000 7f3f0b014700 1 -- 172.21.15.37:0/4036667886 wait complete. 2024-09-16T07:19:19.347 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T07:19:19.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T07:19:19.347 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T07:19:19.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient: get_monmap_and_config 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient: build_initial_monmap 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient: monmap: 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:19.469018+0000 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:19.469018+0000 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:19.466 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 1 Processor -- start 2024-09-16T07:19:19.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 1 -- start start 2024-09-16T07:19:19.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient: init 2024-09-16T07:19:19.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9580a2750 addr v1:172.21.15.37:6789/0 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9580a3240 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff95809d580 0x7ff95813ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient(hunting): picked mon.noname-a con 0x7ff95809d580 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff95809c6c0 con 0x7ff9580a2750 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff95809c9a0 con 0x7ff9580a3240 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95809b130 con 0x7ff95809d580 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95e70a700 10 monclient(hunting): authenticate will time out at 879.955040s 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95d708700 1 -- >> v1:172.21.15.37:6789/0 conn(0x7ff9580a2750 legacy=0x7ff9580a2b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:19:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95df09700 1 -- >> v2:172.21.15.37:3300/0 conn(0x7ff95809d580 msgr2=0x7ff95813ab90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95df09700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff95809d580 0x7ff95813ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95cf07700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ff9580a3240 0x7ff958137460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.37:47384/0 (socket says 172.21.15.37:47384) 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.464+0000 7ff95cf07700 1 -- 172.21.15.37:0/4098581691 learned_addr learned my addr 172.21.15.37:0/4098581691 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831865342 0 0) 0x7ff95809c9a0 con 0x7ff9580a3240 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): _negotiate 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): my global_id is 4462 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9440035d0 con 0x7ff9580a3240 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 533931015 0 0) 0x7ff9440035d0 con 0x7ff9580a3240 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 >> v2:172.21.15.37:3300/0 conn(0x7ff95809d580 msgr2=0x7ff95813ab90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 --2- 172.21.15.37:0/4098581691 >> v2:172.21.15.37:3300/0 conn(0x7ff95809d580 0x7ff95813ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 >> v1:172.21.15.37:6789/0 conn(0x7ff9580a2750 legacy=0x7ff9580a2b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: _finish_hunting 0 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 monclient: found mon.noname-c-legacy 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95809c3e0 con 0x7ff9580a3240 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: _finish_auth 0 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: _check_auth_tickets 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 120013200 0 0) 0x7ff9540019f0 con 0x7ff9580a3240 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 5 monclient: authenticate success, global_id 4462 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T07:19:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: dump: 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 >> v1:172.21.15.143:6789/0 conn(0x7ff9580a3240 legacy=0x7ff958137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff944003a20 0x7ff944003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): picked mon.b con 0x7ff944003a20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff944004870 0x7ff944004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): picked mon.a con 0x7ff944004870 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9440053e0 0x7ff94400dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): picked mon.c con 0x7ff9440053e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95809b130 con 0x7ff944004870 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94400e540 con 0x7ff944003a20 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94400e700 con 0x7ff9440053e0 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient(hunting): _finish_auth 0 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95cf07700 1 -- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff944004870 msgr2=0x7ff944004cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95cf07700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff944004870 0x7ff944004cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9440053e0 0x7ff94400dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95d708700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff944003a20 0x7ff944003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient(hunting): get_auth_request con 0x7ff9440053e0 auth_method 0 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95d708700 10 monclient(hunting): get_auth_request con 0x7ff944003a20 auth_method 0 2024-09-16T07:19:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95d708700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95d708700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95d708700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95d708700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95d708700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95d708700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient(hunting): handle_auth_done global_id 4462 payload 293 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 1 -- 172.21.15.37:0/4098581691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff944003a20 msgr2=0x7ff944003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff944003a20 0x7ff944003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 1 -- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff944004870 msgr2=0x7ff944004cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff944004870 0x7ff944004cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient: _finish_hunting 0 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 1 monclient: found mon.c 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 1 -- 172.21.15.37:0/4098581691 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440035d0 con 0x7ff9440053e0 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient: _finish_auth 0 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 10 monclient: _check_auth_tickets 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 1 -- 172.21.15.37:0/4098581691 reap_dead start 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95df09700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9440053e0 0x7ff94400dc60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff950009e30 tx=0x7ff9440058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3011b590750821e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9500098d0 con 0x7ff9440053e0 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: dump: 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95000bd60 con 0x7ff9440053e0 2024-09-16T07:19:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 1 -- 172.21.15.37:0/4098581691 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950003c70 con 0x7ff9440053e0 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff94ffff700 10 monclient: dump: 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 4 set_mon_vals no callback set 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 10 monclient: shutdown 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 1 -- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9440053e0 msgr2=0x7ff94400dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9440053e0 0x7ff94400dc60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff950009e30 tx=0x7ff9440058a0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 1 -- 172.21.15.37:0/4098581691 shutdown_connections 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 1 --2- 172.21.15.37:0/4098581691 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9440053e0 0x7ff94400dc60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 1 -- 172.21.15.37:0/4098581691 >> 172.21.15.37:0/4098581691 conn(0x7ff9580a3ea0 msgr2=0x7ff9580a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 1 -- 172.21.15.37:0/4098581691 shutdown_connections 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 1 -- 172.21.15.37:0/4098581691 wait complete. 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 10 monclient: build_initial_monmap 2024-09-16T07:19:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.468+0000 7ff95e70a700 10 monclient: monmap: 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 Processor -- start 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 -- start start 2024-09-16T07:19:19.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient: init 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9440053e0 0x7ff9581c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient(hunting): picked mon.noname-a con 0x7ff9440053e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9581305d0 0x7ff9581309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient(hunting): picked mon.noname-c con 0x7ff9581305d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff95811bff0 0x7ff95811c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient(hunting): picked mon.noname-b con 0x7ff95811bff0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95d708700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9440053e0 msgr2=0x7ff9581c5730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:19.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9580724c0 con 0x7ff9440053e0 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95d708700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9440053e0 0x7ff9581c5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9581c5de0 con 0x7ff95811bff0 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95811dc50 con 0x7ff9581305d0 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient(hunting): authenticate will time out at 879.961250s 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9581305d0 0x7ff9581309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9581305d0 0x7ff9581309f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:42040/0 (socket says 172.21.15.37:42040) 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 1 -- 172.21.15.37:0/318467160 learned_addr learned my addr 172.21.15.37:0/318467160 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 10 monclient(hunting): get_auth_request con 0x7ff9581305d0 auth_method 0 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95df09700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff95811bff0 0x7ff95811c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95df09700 10 monclient(hunting): get_auth_request con 0x7ff95811bff0 auth_method 0 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95df09700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95df09700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95df09700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 1 -- 172.21.15.37:0/318467160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff95811bff0 msgr2=0x7ff95811c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff95811bff0 0x7ff95811c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 1 -- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9440053e0 msgr2=0x7ff9581c5730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:19:19.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9440053e0 0x7ff9581c5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 10 monclient: _finish_hunting 0 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 1 monclient: found mon.noname-c 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 1 -- 172.21.15.37:0/318467160 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95811f710 con 0x7ff9581305d0 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95cf07700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9581305d0 0x7ff9581309f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff9540020d0 tx=0x7ff95400a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95400adb0 con 0x7ff9581305d0 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 10 monclient: dump: 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.494 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 10 monclient: _finish_auth 0 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 10 monclient: _check_auth_tickets 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 5 monclient: authenticate success, global_id 4400 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9540070d0 con 0x7ff9581305d0 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient: _renew_subs 2024-09-16T07:19:19.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 -- 172.21.15.37:0/318467160 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95811f9f0 con 0x7ff9581305d0 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954007480 con 0x7ff9581305d0 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 10 monclient: dump: 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.496 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient: _renew_subs 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 -- 172.21.15.37:0/318467160 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff958130f30 con 0x7ff9581305d0 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94effd700 4 set_mon_vals no callback set 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7ff9540075e0 con 0x7ff9581305d0 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95811c990 0x7ff95806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95e70a700 1 -- 172.21.15.37:0/318467160 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff958067530 con 0x7ff95811c990 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff93c0355b0 0x7ff93c037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95d708700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95811c990 0x7ff95806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95d708700 10 monclient: get_auth_request con 0x7ff95811c990 auth_method 0 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95d708700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95d708700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff95d708700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.472+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4738+0+0 (secure 0 0 0) 0x7ff954047a50 con 0x7ff9581305d0 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff95d708700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff95d708700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff95d708700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff95df09700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff93c0355b0 0x7ff93c037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff95df09700 10 monclient: get_auth_request con 0x7ff93c0355b0 auth_method 0 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff95d708700 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff95d708700 1 -- 172.21.15.37:0/318467160 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9580719d0 con 0x7ff95811c990 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff95d708700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95811c990 0x7ff95806e920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff94800e9a0 tx=0x7ff94800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff95df09700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff93c0355b0 0x7ff93c037a70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff94400e890 tx=0x7ff950005000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948003e70 con 0x7ff95811c990 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 10 monclient: dump: 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.499 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ff94800c790 con 0x7ff95811c990 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95811c990 msgr2=0x7ff95806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95811c990 0x7ff95806e920 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff94800e9a0 tx=0x7ff94800ed60 comp rx=0 tx=0).stop 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95811c990 msgr2=0x7ff95806e920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.476+0000 7ff94dffb700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95811c990 0x7ff95806e920 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff94800e9a0 tx=0x7ff94800ed60 comp rx=0 tx=0).stop 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95e70a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95e70a700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95806ee60 0x7ff95806f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95e70a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95e70a700 1 -- 172.21.15.37:0/318467160 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95811fb80 con 0x7ff95806ee60 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95d708700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95806ee60 0x7ff95806f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95d708700 10 monclient: get_auth_request con 0x7ff95806ee60 auth_method 0 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95d708700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95d708700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95d708700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95d708700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95d708700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95d708700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.484+0000 7ff95d708700 10 monclient(hunting): handle_auth_done global_id 4406 payload 824 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95d708700 1 -- 172.21.15.37:0/318467160 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff95811ff10 con 0x7ff95806ee60 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95d708700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95806ee60 0x7ff95806f270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff94800e9a0 tx=0x7ff948005b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948022070 con 0x7ff95806ee60 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 10 monclient: dump: 2024-09-16T07:19:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4608 (secure 0 0 0) 0x7ff94801da30 con 0x7ff95806ee60 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95806ee60 msgr2=0x7ff95806f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95806ee60 0x7ff95806f270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff94800e9a0 tx=0x7ff948005b30 comp rx=0 tx=0).stop 2024-09-16T07:19:19.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 1 -- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95806ee60 msgr2=0x7ff95806f270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:19:19.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff94dffb700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff95806ee60 0x7ff95806f270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.503 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:19:19.503 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:19:19.503 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:19:19.503 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 14, 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:19.504 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:19.505 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:19:19.506 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:19.507 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.508 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:19:19.509 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:19:19.509 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:19.509 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:19.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.509 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:19.509 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:19:19.509 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.509 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.510 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.511 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.512 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:19.513 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.514 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.515 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:19:19.516 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:19:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 1 -- 172.21.15.37:0/318467160 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff93c0355b0 msgr2=0x7ff93c037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff93c0355b0 0x7ff93c037a70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff94400e890 tx=0x7ff950005000 comp rx=0 tx=0).stop 2024-09-16T07:19:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 10 monclient: shutdown 2024-09-16T07:19:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95df09700 1 -- 172.21.15.37:0/318467160 reap_dead start 2024-09-16T07:19:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 1 -- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9581305d0 msgr2=0x7ff9581309f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9581305d0 0x7ff9581309f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff9540020d0 tx=0x7ff95400a340 comp rx=0 tx=0).stop 2024-09-16T07:19:19.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 1 -- 172.21.15.37:0/318467160 shutdown_connections 2024-09-16T07:19:19.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 1 --2- 172.21.15.37:0/318467160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9581305d0 0x7ff9581309f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 1 -- 172.21.15.37:0/318467160 >> 172.21.15.37:0/318467160 conn(0x7ff9580a3ea0 msgr2=0x7ff95809e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:19.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 1 -- 172.21.15.37:0/318467160 shutdown_connections 2024-09-16T07:19:19.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.488+0000 7ff95e70a700 1 -- 172.21.15.37:0/318467160 wait complete. 2024-09-16T07:19:19.518 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T07:19:19.518 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T07:19:19.518 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T07:19:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient: get_monmap_and_config 2024-09-16T07:19:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient: build_initial_monmap 2024-09-16T07:19:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient: monmap: 2024-09-16T07:19:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:19:19.641176+0000 2024-09-16T07:19:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:19:19.641176+0000 2024-09-16T07:19:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:19:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:19:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:19:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:19:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:19:19.638 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 1 Processor -- start 2024-09-16T07:19:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 1 -- start start 2024-09-16T07:19:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient: init 2024-09-16T07:19:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c880a0e90 addr v1:172.21.15.143:6789/0 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4c880a04d0 0x7f4c880a1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient(hunting): picked mon.noname-a con 0x7f4c880a04d0 addr v2:172.21.15.37:3300/0 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c880a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient(hunting): picked mon.noname-b con 0x7f4c880a2d50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c880a0900 con 0x7f4c880a0e90 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c880a17b0 con 0x7f4c880a04d0 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8806cf50 con 0x7f4c880a2d50 2024-09-16T07:19:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8df85700 10 monclient(hunting): authenticate will time out at 880.127175s 2024-09-16T07:19:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c87fff700 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f4c880a04d0 msgr2=0x7f4c880a1270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c87fff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4c880a04d0 0x7f4c880a1270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8d784700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c880a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8d784700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c880a3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:42072/0 (socket says 172.21.15.37:42072) 2024-09-16T07:19:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8d784700 1 -- 172.21.15.37:0/456354373 learned_addr learned my addr 172.21.15.37:0/456354373 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8d784700 10 monclient(hunting): get_auth_request con 0x7f4c880a2d50 auth_method 0 2024-09-16T07:19:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8d784700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8d784700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8d784700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8d784700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8d784700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.636+0000 7f4c8d784700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8d784700 10 monclient(hunting): handle_auth_done global_id 4409 payload 274 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8d784700 1 -- 172.21.15.37:0/456354373 >> v2:172.21.15.37:3300/0 conn(0x7f4c880a04d0 msgr2=0x7f4c880a1270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 1 -- 172.21.15.37:0/456354373 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325507880 0 0) 0x7f4c880a0900 con 0x7f4c880a0e90 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8d784700 1 --2- 172.21.15.37:0/456354373 >> v2:172.21.15.37:3300/0 conn(0x7f4c880a04d0 0x7f4c880a1270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8d784700 1 -- 172.21.15.37:0/456354373 >> v1:172.21.15.143:6789/0 conn(0x7f4c880a0e90 legacy=0x7f4c880a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8d784700 10 monclient: _finish_hunting 0 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8d784700 1 monclient: found mon.noname-b 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8d784700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8d784700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8d784700 1 -- 172.21.15.37:0/456354373 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c880a36c0 con 0x7f4c880a2d50 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8d784700 1 --2- 172.21.15.37:0/456354373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c880a3180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009e50 tx=0x7f4c8000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54d430f69a65a85d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 1 -- 172.21.15.37:0/456354373 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8000e070 con 0x7f4c880a2d50 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 10 monclient: dump: 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 10 monclient: _finish_auth 0 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 10 monclient: _check_auth_tickets 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 1 -- 172.21.15.37:0/456354373 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8000b9c0 con 0x7f4c880a2d50 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 1 -- 172.21.15.37:0/456354373 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8000bd70 con 0x7f4c880a2d50 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c877fe700 10 monclient: dump: 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 5 monclient: authenticate success, global_id 4409 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 10 monclient: get_monmap_and_config success 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 4 set_mon_vals no callback set 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 10 monclient: shutdown 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 1 -- 172.21.15.37:0/456354373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 msgr2=0x7f4c880a3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 1 --2- 172.21.15.37:0/456354373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c880a3180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009e50 tx=0x7f4c8000b330 comp rx=0 tx=0).stop 2024-09-16T07:19:19.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 1 -- 172.21.15.37:0/456354373 shutdown_connections 2024-09-16T07:19:19.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 1 --2- 172.21.15.37:0/456354373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c880a3180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 1 --2- 172.21.15.37:0/456354373 >> v2:172.21.15.37:3300/0 conn(0x7f4c880a04d0 0x7f4c880a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 1 -- 172.21.15.37:0/456354373 >> 172.21.15.37:0/456354373 conn(0x7f4c880a4740 msgr2=0x7f4c880a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:19.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 1 -- 172.21.15.37:0/456354373 shutdown_connections 2024-09-16T07:19:19.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 1 -- 172.21.15.37:0/456354373 wait complete. 2024-09-16T07:19:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 10 monclient: build_initial_monmap 2024-09-16T07:19:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 10 monclient: monmap: 2024-09-16T07:19:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:19:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 1 Processor -- start 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 1 -- start start 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.640+0000 7f4c8df85700 10 monclient: init 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient: _reopen_session rank -1 2024-09-16T07:19:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c880a04d0 0x7f4c881c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient(hunting): picked mon.noname-b con 0x7f4c880a04d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c880a0e90 0x7f4c881c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient(hunting): picked mon.noname-a con 0x7f4c880a0e90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c881c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient(hunting): picked mon.noname-c con 0x7f4c880a2d50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c881aaf10 con 0x7f4c880a0e90 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8809df30 con 0x7f4c880a04d0 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c87fff700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c880a0e90 msgr2=0x7f4c881c3e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8806d780 con 0x7f4c880a2d50 2024-09-16T07:19:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient(hunting): _renew_subs 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient(hunting): authenticate will time out at 880.132385s 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c87fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c880a0e90 0x7f4c881c3e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c881c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c881c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:42078/0 (socket says 172.21.15.37:42078) 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 1 -- 172.21.15.37:0/623139743 learned_addr learned my addr 172.21.15.37:0/623139743 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 10 monclient(hunting): get_auth_request con 0x7f4c880a2d50 auth_method 0 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:19.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c880a04d0 0x7f4c881c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient(hunting): get_auth_request con 0x7f4c880a04d0 auth_method 0 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 1 -- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c880a04d0 msgr2=0x7f4c881c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c880a04d0 0x7f4c881c68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 1 -- 172.21.15.37:0/623139743 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c880a0e90 msgr2=0x7f4c881c3e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c880a0e90 0x7f4c881c3e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 10 monclient: _finish_hunting 0 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 1 monclient: found mon.noname-c 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:19:19.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 1 -- 172.21.15.37:0/623139743 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c80009b30 con 0x7f4c880a2d50 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c880a04d0 0x7f4c881c68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8d784700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c881c9e00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c80013de0 tx=0x7f4c80004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8001c640 con 0x7f4c880a2d50 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 10 monclient: dump: 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.659 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.660 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.660 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.660 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 10 monclient: _finish_auth 0 2024-09-16T07:19:19.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 10 monclient: _check_auth_tickets 2024-09-16T07:19:19.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:19:19.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8001c7c0 con 0x7f4c880a2d50 2024-09-16T07:19:19.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 5 monclient: authenticate success, global_id 4412 2024-09-16T07:19:19.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient: _renew_subs 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 1 -- 172.21.15.37:0/623139743 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c881a8cd0 con 0x7f4c880a2d50 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8001cb70 con 0x7f4c880a2d50 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c86ffd700 4 set_mon_vals no callback set 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 10 monclient: dump: 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.662 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient: _renew_subs 2024-09-16T07:19:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:19:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 1 -- 172.21.15.37:0/623139743 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c881ab7b0 con 0x7f4c880a2d50 2024-09-16T07:19:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f4c8001ccd0 con 0x7f4c880a2d50 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881ca340 0x7f4c881ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8df85700 1 -- 172.21.15.37:0/623139743 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88067400 con 0x7f4c881ca340 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4c6c0355b0 0x7f4c6c037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881ca340 0x7f4c881ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient: get_auth_request con 0x7f4c881ca340 auth_method 0 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f4c80049b50 con 0x7f4c880a2d50 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.644+0000 7f4c8cf83700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c87fff700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4c6c0355b0 0x7f4c6c037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c87fff700 10 monclient: get_auth_request con 0x7f4c6c0355b0 auth_method 0 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c8cf83700 10 monclient(hunting): handle_auth_done global_id 4474 payload 824 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c8cf83700 1 -- 172.21.15.37:0/623139743 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c8806cbd0 con 0x7f4c881ca340 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c8cf83700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881ca340 0x7f4c881ca720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4c740011e0 tx=0x7f4c7400ea90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c87fff700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4c6c0355b0 0x7f4c6c037a70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c006fd0 tx=0x7f4c7c006de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c740052e0 con 0x7f4c881ca340 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 10 monclient: dump: 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.665 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f4c74005440 con 0x7f4c881ca340 2024-09-16T07:19:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:19:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:19:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881ca340 msgr2=0x7f4c881ca720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881ca340 0x7f4c881ca720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4c740011e0 tx=0x7f4c7400ea90 comp rx=0 tx=0).stop 2024-09-16T07:19:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881ca340 msgr2=0x7f4c881ca720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:19:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.648+0000 7f4c857fa700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881ca340 0x7f4c881ca720 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8df85700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8df85700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881a3fb0 0x7f4c881a43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8df85700 10 monclient(hunting): start opening mon connection 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8df85700 1 -- 172.21.15.37:0/623139743 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8809f530 con 0x7f4c881a3fb0 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8cf83700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881a3fb0 0x7f4c881a43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8cf83700 10 monclient: get_auth_request con 0x7f4c881a3fb0 auth_method 0 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8cf83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8cf83700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8cf83700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8cf83700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8cf83700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:19:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.656+0000 7f4c8cf83700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8cf83700 10 monclient(hunting): handle_auth_done global_id 4477 payload 824 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8cf83700 1 -- 172.21.15.37:0/623139743 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4c880a54d0 con 0x7f4c881a3fb0 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8cf83700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881a3fb0 0x7f4c881a43c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c740011e0 tx=0x7f4c74001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74021070 con 0x7f4c881a3fb0 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 10 monclient: dump: 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:19:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4776 (secure 0 0 0) 0x7f4c740038b0 con 0x7f4c881a3fb0 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881a3fb0 msgr2=0x7f4c881a43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881a3fb0 0x7f4c881a43c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c740011e0 tx=0x7f4c74001080 comp rx=0 tx=0).stop 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 1 -- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881a3fb0 msgr2=0x7f4c881a43c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:19:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c857fa700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c881a3fb0 0x7f4c881a43c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 14, 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:19:19.671 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.672 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.673 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:19:19.674 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:19:19.675 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.678 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:19:19.679 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:19:19.680 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.681 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.682 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:19:19.683 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-09-16T07:19:19.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:19:19.683 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:19:19.683 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:19:19.683 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:19:19.683 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:19:19.683 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:19:19.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:19:19.683 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:19:19.683 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:19:19.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8df85700 1 -- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4c6c0355b0 msgr2=0x7f4c6c037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8df85700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4c6c0355b0 0x7f4c6c037a70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c006fd0 tx=0x7f4c7c006de0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8df85700 10 monclient: shutdown 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8df85700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8df85700 1 -- 172.21.15.37:0/623139743 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 msgr2=0x7f4c881c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8df85700 1 --2- 172.21.15.37:0/623139743 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4c880a2d50 0x7f4c881c9e00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c80013de0 tx=0x7f4c80004af0 comp rx=0 tx=0).stop 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8d784700 1 -- 172.21.15.37:0/623139743 reap_dead start 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8d784700 1 -- 172.21.15.37:0/623139743 reap_dead start 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8df85700 1 -- 172.21.15.37:0/623139743 shutdown_connections 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8df85700 1 -- 172.21.15.37:0/623139743 >> 172.21.15.37:0/623139743 conn(0x7f4c880a4740 msgr2=0x7f4c8809c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8df85700 1 -- 172.21.15.37:0/623139743 shutdown_connections 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:19:19.660+0000 7f4c8df85700 1 -- 172.21.15.37:0/623139743 wait complete. 2024-09-16T07:19:19.685 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24228]: starting. 2024-09-16T07:19:19.686 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24228]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24226 2024-09-16T07:19:19.686 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24228]: shutting down. 2024-09-16T07:19:19.686 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T07:19:19.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T07:19:21.860 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:21.860 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24261]: starting. 2024-09-16T07:19:21.860 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24261]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24226 2024-09-16T07:19:21.860 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24261]: created object 0... 2024-09-16T07:19:21.861 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24261]: created object 25... 2024-09-16T07:19:21.861 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24261]: created object 49... 2024-09-16T07:19:21.861 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24261]: finishing. 2024-09-16T07:19:21.861 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24261]: shutting down. 2024-09-16T07:19:21.894 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:21.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24263]: starting. 2024-09-16T07:19:21.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24263]: listing objects. 2024-09-16T07:19:21.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24263]: listed object 0... 2024-09-16T07:19:21.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24263]: listed object 25... 2024-09-16T07:19:21.895 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24263]: saw 50 objects 2024-09-16T07:19:21.895 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24263]: shutting down. 2024-09-16T07:19:22.704 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:22.704 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24262]: starting. 2024-09-16T07:19:22.704 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24262]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24226 2024-09-16T07:19:22.704 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24262]: shutting down. 2024-09-16T07:19:22.707 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:22.707 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:22.708 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:19:22.710 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:19:24.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24316]: starting. 2024-09-16T07:19:24.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24316]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24315 2024-09-16T07:19:24.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24316]: created object 0... 2024-09-16T07:19:24.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24316]: created object 25... 2024-09-16T07:19:24.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24316]: created object 49... 2024-09-16T07:19:24.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24316]: finishing. 2024-09-16T07:19:24.875 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24316]: shutting down. 2024-09-16T07:19:25.727 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24317]: starting. 2024-09-16T07:19:25.728 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24317]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24315 2024-09-16T07:19:25.728 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24317]: shutting down. 2024-09-16T07:19:27.902 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:27.902 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24352]: starting. 2024-09-16T07:19:27.902 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24352]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24315 2024-09-16T07:19:27.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24352]: created object 0... 2024-09-16T07:19:27.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24352]: created object 25... 2024-09-16T07:19:27.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24352]: created object 49... 2024-09-16T07:19:27.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24352]: finishing. 2024-09-16T07:19:27.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24352]: shutting down. 2024-09-16T07:19:27.935 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:27.936 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24354]: starting. 2024-09-16T07:19:27.936 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24354]: listing objects. 2024-09-16T07:19:27.936 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24354]: listed object 0... 2024-09-16T07:19:27.936 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24354]: listed object 25... 2024-09-16T07:19:27.936 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24354]: saw 50 objects 2024-09-16T07:19:27.936 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24354]: shutting down. 2024-09-16T07:19:28.760 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:28.761 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24353]: starting. 2024-09-16T07:19:28.761 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24353]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24315 2024-09-16T07:19:28.761 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24353]: shutting down. 2024-09-16T07:19:28.764 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:28.764 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:28.764 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:19:28.768 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:19:30.926 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24407]: starting. 2024-09-16T07:19:30.926 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24407]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24406 2024-09-16T07:19:30.926 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24407]: created object 0... 2024-09-16T07:19:30.927 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24407]: created object 25... 2024-09-16T07:19:30.927 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24407]: created object 49... 2024-09-16T07:19:30.927 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24407]: finishing. 2024-09-16T07:19:30.927 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24407]: shutting down. 2024-09-16T07:19:31.782 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24408]: starting. 2024-09-16T07:19:31.782 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24408]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24406 2024-09-16T07:19:31.782 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24408]: shutting down. 2024-09-16T07:19:33.964 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:33.965 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24443]: starting. 2024-09-16T07:19:33.965 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24443]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24406 2024-09-16T07:19:33.965 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24443]: created object 0... 2024-09-16T07:19:33.965 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24443]: created object 25... 2024-09-16T07:19:33.965 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24443]: created object 49... 2024-09-16T07:19:33.965 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24443]: finishing. 2024-09-16T07:19:33.965 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24443]: shutting down. 2024-09-16T07:19:33.995 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:33.995 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24445]: starting. 2024-09-16T07:19:33.996 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24445]: listing objects. 2024-09-16T07:19:33.996 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24445]: listed object 0... 2024-09-16T07:19:33.996 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24445]: listed object 25... 2024-09-16T07:19:33.996 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24445]: saw 50 objects 2024-09-16T07:19:33.996 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24445]: shutting down. 2024-09-16T07:19:34.817 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:34.817 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24444]: starting. 2024-09-16T07:19:34.818 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24444]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24406 2024-09-16T07:19:34.818 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24444]: shutting down. 2024-09-16T07:19:34.820 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:34.820 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:34.820 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:19:34.823 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:19:36.990 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24498]: starting. 2024-09-16T07:19:36.991 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24498]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24497 2024-09-16T07:19:36.991 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24498]: created object 0... 2024-09-16T07:19:36.991 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24498]: created object 25... 2024-09-16T07:19:36.991 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24498]: created object 49... 2024-09-16T07:19:36.991 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24498]: finishing. 2024-09-16T07:19:36.991 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24498]: shutting down. 2024-09-16T07:19:37.837 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24499]: starting. 2024-09-16T07:19:37.837 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24499]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24497 2024-09-16T07:19:37.838 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24499]: shutting down. 2024-09-16T07:19:40.025 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:40.025 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24534]: starting. 2024-09-16T07:19:40.026 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24534]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24497 2024-09-16T07:19:40.026 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24534]: created object 0... 2024-09-16T07:19:40.026 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24534]: created object 25... 2024-09-16T07:19:40.026 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24534]: created object 49... 2024-09-16T07:19:40.026 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24534]: finishing. 2024-09-16T07:19:40.026 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24534]: shutting down. 2024-09-16T07:19:40.061 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:40.061 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24536]: starting. 2024-09-16T07:19:40.061 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24536]: listing objects. 2024-09-16T07:19:40.061 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24536]: listed object 0... 2024-09-16T07:19:40.062 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24536]: listed object 25... 2024-09-16T07:19:40.062 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24536]: saw 50 objects 2024-09-16T07:19:40.062 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24536]: shutting down. 2024-09-16T07:19:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24535]: starting. 2024-09-16T07:19:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24535]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24497 2024-09-16T07:19:40.589 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24535]: shutting down. 2024-09-16T07:19:40.592 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:40.593 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:40.593 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:19:40.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:19:42.771 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24589]: starting. 2024-09-16T07:19:42.771 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24589]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24588 2024-09-16T07:19:42.771 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24589]: created object 0... 2024-09-16T07:19:42.772 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24589]: created object 25... 2024-09-16T07:19:42.772 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24589]: created object 49... 2024-09-16T07:19:42.772 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24589]: finishing. 2024-09-16T07:19:42.772 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24589]: shutting down. 2024-09-16T07:19:43.608 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24590]: starting. 2024-09-16T07:19:43.608 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24590]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24588 2024-09-16T07:19:43.608 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24590]: shutting down. 2024-09-16T07:19:45.783 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:45.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24625]: starting. 2024-09-16T07:19:45.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24625]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24588 2024-09-16T07:19:45.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24625]: created object 0... 2024-09-16T07:19:45.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24625]: created object 25... 2024-09-16T07:19:45.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24625]: created object 49... 2024-09-16T07:19:45.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24625]: finishing. 2024-09-16T07:19:45.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24625]: shutting down. 2024-09-16T07:19:45.816 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:45.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24627]: starting. 2024-09-16T07:19:45.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24627]: listing objects. 2024-09-16T07:19:45.817 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24627]: listed object 0... 2024-09-16T07:19:45.817 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24627]: listed object 25... 2024-09-16T07:19:45.817 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24627]: saw 50 objects 2024-09-16T07:19:45.817 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24627]: shutting down. 2024-09-16T07:19:46.641 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:46.641 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24626]: starting. 2024-09-16T07:19:46.641 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24626]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24588 2024-09-16T07:19:46.641 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24626]: shutting down. 2024-09-16T07:19:46.643 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:46.644 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:46.644 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:19:46.646 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:19:48.835 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24680]: starting. 2024-09-16T07:19:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24680]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24679 2024-09-16T07:19:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24680]: created object 0... 2024-09-16T07:19:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24680]: created object 25... 2024-09-16T07:19:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24680]: created object 49... 2024-09-16T07:19:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24680]: finishing. 2024-09-16T07:19:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24680]: shutting down. 2024-09-16T07:19:49.680 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24681]: starting. 2024-09-16T07:19:49.680 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24681]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24679 2024-09-16T07:19:49.680 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24681]: shutting down. 2024-09-16T07:19:51.736 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:51.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24716]: starting. 2024-09-16T07:19:51.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24716]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24679 2024-09-16T07:19:51.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24716]: created object 0... 2024-09-16T07:19:51.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24716]: created object 25... 2024-09-16T07:19:51.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24716]: created object 49... 2024-09-16T07:19:51.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24716]: finishing. 2024-09-16T07:19:51.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24716]: shutting down. 2024-09-16T07:19:51.767 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:51.767 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24718]: starting. 2024-09-16T07:19:51.767 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24718]: listing objects. 2024-09-16T07:19:51.768 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24718]: listed object 0... 2024-09-16T07:19:51.768 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24718]: listed object 25... 2024-09-16T07:19:51.768 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24718]: saw 50 objects 2024-09-16T07:19:51.768 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24718]: shutting down. 2024-09-16T07:19:52.607 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:52.607 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24717]: starting. 2024-09-16T07:19:52.608 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24717]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24679 2024-09-16T07:19:52.608 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24717]: shutting down. 2024-09-16T07:19:52.610 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:52.610 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:52.610 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:19:52.613 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:19:54.759 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24771]: starting. 2024-09-16T07:19:54.759 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24771]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24770 2024-09-16T07:19:54.760 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24771]: created object 0... 2024-09-16T07:19:54.760 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24771]: created object 25... 2024-09-16T07:19:54.760 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24771]: created object 49... 2024-09-16T07:19:54.760 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24771]: finishing. 2024-09-16T07:19:54.760 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24771]: shutting down. 2024-09-16T07:19:55.627 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24772]: starting. 2024-09-16T07:19:55.627 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24772]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24770 2024-09-16T07:19:55.627 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24772]: shutting down. 2024-09-16T07:19:57.811 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:57.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24807]: starting. 2024-09-16T07:19:57.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24807]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24770 2024-09-16T07:19:57.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24807]: created object 0... 2024-09-16T07:19:57.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24807]: created object 25... 2024-09-16T07:19:57.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24807]: created object 49... 2024-09-16T07:19:57.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24807]: finishing. 2024-09-16T07:19:57.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24807]: shutting down. 2024-09-16T07:19:57.841 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:57.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24809]: starting. 2024-09-16T07:19:57.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24809]: listing objects. 2024-09-16T07:19:57.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24809]: listed object 0... 2024-09-16T07:19:57.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24809]: listed object 25... 2024-09-16T07:19:57.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24809]: saw 50 objects 2024-09-16T07:19:57.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24809]: shutting down. 2024-09-16T07:19:58.659 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:58.659 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24808]: starting. 2024-09-16T07:19:58.659 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24808]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24770 2024-09-16T07:19:58.659 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24808]: shutting down. 2024-09-16T07:19:58.661 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:58.662 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:19:58.662 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:19:58.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:20:00.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24862]: starting. 2024-09-16T07:20:00.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24862]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24861 2024-09-16T07:20:00.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24862]: created object 0... 2024-09-16T07:20:00.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24862]: created object 25... 2024-09-16T07:20:00.737 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24862]: created object 49... 2024-09-16T07:20:00.738 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24862]: finishing. 2024-09-16T07:20:00.738 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24862]: shutting down. 2024-09-16T07:20:01.608 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24863]: starting. 2024-09-16T07:20:01.608 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24863]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24861 2024-09-16T07:20:01.608 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24863]: shutting down. 2024-09-16T07:20:03.783 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:03.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24898]: starting. 2024-09-16T07:20:03.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24898]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24861 2024-09-16T07:20:03.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24898]: created object 0... 2024-09-16T07:20:03.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24898]: created object 25... 2024-09-16T07:20:03.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24898]: created object 49... 2024-09-16T07:20:03.784 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24898]: finishing. 2024-09-16T07:20:03.784 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24898]: shutting down. 2024-09-16T07:20:03.814 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:03.814 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24900]: starting. 2024-09-16T07:20:03.814 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24900]: listing objects. 2024-09-16T07:20:03.815 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24900]: listed object 0... 2024-09-16T07:20:03.815 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24900]: listed object 25... 2024-09-16T07:20:03.815 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24900]: saw 50 objects 2024-09-16T07:20:03.815 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24900]: shutting down. 2024-09-16T07:20:04.631 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:04.632 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24899]: starting. 2024-09-16T07:20:04.632 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24899]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24861 2024-09-16T07:20:04.632 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24899]: shutting down. 2024-09-16T07:20:04.634 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:04.634 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:04.634 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:20:04.638 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:20:06.802 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24955]: starting. 2024-09-16T07:20:06.802 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24955]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24954 2024-09-16T07:20:06.802 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24955]: created object 0... 2024-09-16T07:20:06.802 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24955]: created object 25... 2024-09-16T07:20:06.802 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24955]: created object 49... 2024-09-16T07:20:06.802 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24955]: finishing. 2024-09-16T07:20:06.802 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[24955]: shutting down. 2024-09-16T07:20:07.643 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24956]: starting. 2024-09-16T07:20:07.643 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24956]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24954 2024-09-16T07:20:07.643 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[24956]: shutting down. 2024-09-16T07:20:09.824 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:09.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24991]: starting. 2024-09-16T07:20:09.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24991]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-24954 2024-09-16T07:20:09.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24991]: created object 0... 2024-09-16T07:20:09.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24991]: created object 25... 2024-09-16T07:20:09.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24991]: created object 49... 2024-09-16T07:20:09.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24991]: finishing. 2024-09-16T07:20:09.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[24991]: shutting down. 2024-09-16T07:20:09.857 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:09.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24993]: starting. 2024-09-16T07:20:09.858 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24993]: listing objects. 2024-09-16T07:20:09.858 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24993]: listed object 0... 2024-09-16T07:20:09.858 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24993]: listed object 25... 2024-09-16T07:20:09.858 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24993]: saw 50 objects 2024-09-16T07:20:09.858 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[24993]: shutting down. 2024-09-16T07:20:10.598 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:10.598 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24992]: starting. 2024-09-16T07:20:10.598 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24992]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-24954 2024-09-16T07:20:10.598 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[24992]: shutting down. 2024-09-16T07:20:10.602 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:10.602 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:10.602 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:20:10.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:20:12.767 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25046]: starting. 2024-09-16T07:20:12.767 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25046]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25045 2024-09-16T07:20:12.767 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25046]: created object 0... 2024-09-16T07:20:12.768 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25046]: created object 25... 2024-09-16T07:20:12.768 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25046]: created object 49... 2024-09-16T07:20:12.768 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25046]: finishing. 2024-09-16T07:20:12.768 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25046]: shutting down. 2024-09-16T07:20:13.622 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25047]: starting. 2024-09-16T07:20:13.623 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25047]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25045 2024-09-16T07:20:13.623 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25047]: shutting down. 2024-09-16T07:20:15.799 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:15.799 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25082]: starting. 2024-09-16T07:20:15.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25082]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25045 2024-09-16T07:20:15.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25082]: created object 0... 2024-09-16T07:20:15.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25082]: created object 25... 2024-09-16T07:20:15.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25082]: created object 49... 2024-09-16T07:20:15.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25082]: finishing. 2024-09-16T07:20:15.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25082]: shutting down. 2024-09-16T07:20:15.831 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:15.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25084]: starting. 2024-09-16T07:20:15.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25084]: listing objects. 2024-09-16T07:20:15.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25084]: listed object 0... 2024-09-16T07:20:15.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25084]: listed object 25... 2024-09-16T07:20:15.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25084]: saw 50 objects 2024-09-16T07:20:15.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25084]: shutting down. 2024-09-16T07:20:16.648 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:16.648 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25083]: starting. 2024-09-16T07:20:16.648 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25083]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25045 2024-09-16T07:20:16.648 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25083]: shutting down. 2024-09-16T07:20:16.650 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:16.651 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:16.651 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:20:16.653 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:20:18.942 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25137]: starting. 2024-09-16T07:20:18.943 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25137]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25136 2024-09-16T07:20:18.943 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25137]: created object 0... 2024-09-16T07:20:18.943 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25137]: created object 25... 2024-09-16T07:20:18.943 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25137]: created object 49... 2024-09-16T07:20:18.943 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25137]: finishing. 2024-09-16T07:20:18.943 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25137]: shutting down. 2024-09-16T07:20:19.684 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25138]: starting. 2024-09-16T07:20:19.684 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25138]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25136 2024-09-16T07:20:19.684 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25138]: shutting down. 2024-09-16T07:20:21.764 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:21.764 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25173]: starting. 2024-09-16T07:20:21.764 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25173]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25136 2024-09-16T07:20:21.764 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25173]: created object 0... 2024-09-16T07:20:21.764 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25173]: created object 25... 2024-09-16T07:20:21.764 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25173]: created object 49... 2024-09-16T07:20:21.764 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25173]: finishing. 2024-09-16T07:20:21.764 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25173]: shutting down. 2024-09-16T07:20:21.794 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:21.794 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25175]: starting. 2024-09-16T07:20:21.794 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25175]: listing objects. 2024-09-16T07:20:21.794 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25175]: listed object 0... 2024-09-16T07:20:21.794 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25175]: listed object 25... 2024-09-16T07:20:21.795 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25175]: saw 50 objects 2024-09-16T07:20:21.795 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25175]: shutting down. 2024-09-16T07:20:22.622 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:22.623 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25174]: starting. 2024-09-16T07:20:22.623 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25174]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25136 2024-09-16T07:20:22.623 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25174]: shutting down. 2024-09-16T07:20:22.623 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:22.623 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:22.624 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:20:22.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:20:24.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25228]: starting. 2024-09-16T07:20:24.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25228]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25227 2024-09-16T07:20:24.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25228]: created object 0... 2024-09-16T07:20:24.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25228]: created object 25... 2024-09-16T07:20:24.790 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25228]: created object 49... 2024-09-16T07:20:24.790 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25228]: finishing. 2024-09-16T07:20:24.790 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25228]: shutting down. 2024-09-16T07:20:25.651 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25229]: starting. 2024-09-16T07:20:25.651 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25229]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25227 2024-09-16T07:20:25.651 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25229]: shutting down. 2024-09-16T07:20:27.815 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:27.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25264]: starting. 2024-09-16T07:20:27.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25264]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25227 2024-09-16T07:20:27.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25264]: created object 0... 2024-09-16T07:20:27.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25264]: created object 25... 2024-09-16T07:20:27.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25264]: created object 49... 2024-09-16T07:20:27.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25264]: finishing. 2024-09-16T07:20:27.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25264]: shutting down. 2024-09-16T07:20:28.056 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:28.056 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25266]: starting. 2024-09-16T07:20:28.057 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25266]: listing objects. 2024-09-16T07:20:28.057 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25266]: listed object 0... 2024-09-16T07:20:28.057 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25266]: listed object 25... 2024-09-16T07:20:28.057 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25266]: saw 50 objects 2024-09-16T07:20:28.057 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25266]: shutting down. 2024-09-16T07:20:28.677 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:28.678 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25265]: starting. 2024-09-16T07:20:28.678 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25265]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25227 2024-09-16T07:20:28.678 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25265]: shutting down. 2024-09-16T07:20:28.679 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:28.679 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:28.679 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:20:28.682 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:20:30.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25319]: starting. 2024-09-16T07:20:30.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25319]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25318 2024-09-16T07:20:30.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25319]: created object 0... 2024-09-16T07:20:30.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25319]: created object 25... 2024-09-16T07:20:30.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25319]: created object 49... 2024-09-16T07:20:30.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25319]: finishing. 2024-09-16T07:20:30.790 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25319]: shutting down. 2024-09-16T07:20:31.607 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25320]: starting. 2024-09-16T07:20:31.607 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25320]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25318 2024-09-16T07:20:31.607 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25320]: shutting down. 2024-09-16T07:20:33.774 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:33.774 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25355]: starting. 2024-09-16T07:20:33.774 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25355]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25318 2024-09-16T07:20:33.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25355]: created object 0... 2024-09-16T07:20:33.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25355]: created object 25... 2024-09-16T07:20:33.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25355]: created object 49... 2024-09-16T07:20:33.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25355]: finishing. 2024-09-16T07:20:33.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25355]: shutting down. 2024-09-16T07:20:33.808 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:33.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25357]: starting. 2024-09-16T07:20:33.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25357]: listing objects. 2024-09-16T07:20:33.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25357]: listed object 0... 2024-09-16T07:20:33.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25357]: listed object 25... 2024-09-16T07:20:33.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25357]: saw 50 objects 2024-09-16T07:20:33.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25357]: shutting down. 2024-09-16T07:20:34.629 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:34.629 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25356]: starting. 2024-09-16T07:20:34.629 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25356]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25318 2024-09-16T07:20:34.629 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25356]: shutting down. 2024-09-16T07:20:34.632 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:34.632 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:34.632 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:20:34.634 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:20:36.794 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25410]: starting. 2024-09-16T07:20:36.794 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25410]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25409 2024-09-16T07:20:36.794 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25410]: created object 0... 2024-09-16T07:20:36.795 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25410]: created object 25... 2024-09-16T07:20:36.795 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25410]: created object 49... 2024-09-16T07:20:36.795 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25410]: finishing. 2024-09-16T07:20:36.795 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25410]: shutting down. 2024-09-16T07:20:37.636 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25411]: starting. 2024-09-16T07:20:37.637 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25411]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25409 2024-09-16T07:20:37.637 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25411]: shutting down. 2024-09-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25446]: starting. 2024-09-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25446]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25409 2024-09-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25446]: created object 0... 2024-09-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25446]: created object 25... 2024-09-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25446]: created object 49... 2024-09-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25446]: finishing. 2024-09-16T07:20:39.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25446]: shutting down. 2024-09-16T07:20:39.850 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:39.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25448]: starting. 2024-09-16T07:20:39.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25448]: listing objects. 2024-09-16T07:20:39.851 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25448]: listed object 0... 2024-09-16T07:20:39.851 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25448]: listed object 25... 2024-09-16T07:20:39.851 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25448]: saw 50 objects 2024-09-16T07:20:39.851 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25448]: shutting down. 2024-09-16T07:20:40.609 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:40.609 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25447]: starting. 2024-09-16T07:20:40.609 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25447]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25409 2024-09-16T07:20:40.609 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25447]: shutting down. 2024-09-16T07:20:40.611 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:40.611 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:40.611 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:20:40.613 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:20:42.785 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25501]: starting. 2024-09-16T07:20:42.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25501]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25500 2024-09-16T07:20:42.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25501]: created object 0... 2024-09-16T07:20:42.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25501]: created object 25... 2024-09-16T07:20:42.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25501]: created object 49... 2024-09-16T07:20:42.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25501]: finishing. 2024-09-16T07:20:42.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25501]: shutting down. 2024-09-16T07:20:43.641 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25502]: starting. 2024-09-16T07:20:43.641 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25502]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25500 2024-09-16T07:20:43.641 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25502]: shutting down. 2024-09-16T07:20:45.821 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:45.821 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25537]: starting. 2024-09-16T07:20:45.821 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25537]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25500 2024-09-16T07:20:45.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25537]: created object 0... 2024-09-16T07:20:45.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25537]: created object 25... 2024-09-16T07:20:45.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25537]: created object 49... 2024-09-16T07:20:45.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25537]: finishing. 2024-09-16T07:20:45.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25537]: shutting down. 2024-09-16T07:20:45.851 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:45.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25539]: starting. 2024-09-16T07:20:45.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25539]: listing objects. 2024-09-16T07:20:45.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25539]: listed object 0... 2024-09-16T07:20:45.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25539]: listed object 25... 2024-09-16T07:20:45.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25539]: saw 50 objects 2024-09-16T07:20:45.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25539]: shutting down. 2024-09-16T07:20:46.674 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:46.675 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25538]: starting. 2024-09-16T07:20:46.675 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25538]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25500 2024-09-16T07:20:46.675 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25538]: shutting down. 2024-09-16T07:20:46.676 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:46.677 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:46.677 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:20:46.679 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:20:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25592]: starting. 2024-09-16T07:20:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25592]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25591 2024-09-16T07:20:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25592]: created object 0... 2024-09-16T07:20:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25592]: created object 25... 2024-09-16T07:20:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25592]: created object 49... 2024-09-16T07:20:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25592]: finishing. 2024-09-16T07:20:48.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25592]: shutting down. 2024-09-16T07:20:49.687 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T07:20:49.687 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T07:20:49.687 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T07:20:49.688 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T07:20:49.690 INFO:tasks.ceph.mon.a:Started 2024-09-16T07:20:49.690 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T07:20:49.691 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:20:49.692 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25593]: starting. 2024-09-16T07:20:49.692 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25593]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25591 2024-09-16T07:20:49.692 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25593]: shutting down. 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.804+0000 7fc47a7d6700 10 monclient: get_monmap_and_config 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.804+0000 7fc47a7d6700 10 monclient: build_initial_monmap 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.804+0000 7fc47a7d6700 10 monclient: monmap: 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:49.809517+0000 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:49.809517+0000 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:49.808 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:49.809 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:49.809 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:49.809 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:49.809 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:49.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.804+0000 7fc47a7d6700 1 Processor -- start 2024-09-16T07:20:49.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.804+0000 7fc47a7d6700 1 -- start start 2024-09-16T07:20:49.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.804+0000 7fc47a7d6700 10 monclient: init 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc47406d430 addr v1:172.21.15.143:6789/0 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc4740a1050 0x7fc4741375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 10 monclient(hunting): picked mon.noname-a con 0x7fc4740a1050 addr v2:172.21.15.37:3300/0 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc47406df20 addr v1:172.21.15.37:6789/0 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47409ee80 con 0x7fc47406df20 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47406f980 con 0x7fc47406d430 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4740a0dd0 con 0x7fc4740a1050 2024-09-16T07:20:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 10 monclient(hunting): authenticate will time out at 970.297124s 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc4740a1050 0x7fc4741375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc47406df20 0x7fc474139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33462/0 (socket says 172.21.15.37:33462) 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 1 -- 172.21.15.37:0/3672900931 learned_addr learned my addr 172.21.15.37:0/3672900931 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): get_auth_request con 0x7fc4740a1050 auth_method 0 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 -- 172.21.15.37:0/3672900931 >> v2:172.21.15.37:3300/0 conn(0x7fc4740a1050 msgr2=0x7fc4741375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 -- 172.21.15.37:0/3672900931 >> v2:172.21.15.37:3300/0 conn(0x7fc4740a1050 msgr2=0x7fc4741375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 --2- 172.21.15.37:0/3672900931 >> v2:172.21.15.37:3300/0 conn(0x7fc4740a1050 0x7fc4741375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 --2- 172.21.15.37:0/3672900931 >> v2:172.21.15.37:3300/0 conn(0x7fc4740a1050 0x7fc4741375d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:20:49.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782321346 0 0) 0x7fc47406f980 con 0x7fc47406d430 2024-09-16T07:20:49.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): _negotiate 2024-09-16T07:20:49.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:49.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:49.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): my global_id is 4966 2024-09-16T07:20:49.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4600035d0 con 0x7fc47406d430 2024-09-16T07:20:49.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 466984422 0 0) 0x7fc4600035d0 con 0x7fc47406d430 2024-09-16T07:20:49.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 >> v2:172.21.15.37:3300/0 conn(0x7fc4740a1050 msgr2=0x7fc4741375d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:20:49.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 --2- 172.21.15.37:0/3672900931 >> v2:172.21.15.37:3300/0 conn(0x7fc4740a1050 0x7fc4741375d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:49.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 >> v1:172.21.15.37:6789/0 conn(0x7fc47406df20 legacy=0x7fc474139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:49.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient: _finish_hunting 0 2024-09-16T07:20:49.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 monclient: found mon.noname-c-legacy 2024-09-16T07:20:49.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:49.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T07:20:49.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47406f0e0 con 0x7fc47406d430 2024-09-16T07:20:49.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient: _finish_auth 0 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient: _check_auth_tickets 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 120013200 0 0) 0x7fc470001670 con 0x7fc47406d430 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 5 monclient: authenticate success, global_id 4966 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient: dump: 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:49.819 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 >> v1:172.21.15.143:6789/0 conn(0x7fc47406d430 legacy=0x7fc47406d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc460003a20 0x7fc460003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:49.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): picked mon.a con 0x7fc460003a20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc460004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): picked mon.c con 0x7fc460004870 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc460005430 0x7fc46000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): picked mon.b con 0x7fc460005430 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4740a0dd0 con 0x7fc460003a20 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46000c2c0 con 0x7fc460005430 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46000c480 con 0x7fc460004870 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc46bfff700 10 monclient(hunting): _finish_auth 0 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc47a7d6700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:49.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc460004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:49.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): get_auth_request con 0x7fc460004870 auth_method 0 2024-09-16T07:20:49.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:49.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:49.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:49.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc4797d4700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc460003a20 0x7fc460003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:49.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc460005430 0x7fc46000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:49.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc4797d4700 10 monclient(hunting): get_auth_request con 0x7fc460003a20 auth_method 0 2024-09-16T07:20:49.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc4797d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc4797d4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc4797d4700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:20:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 10 monclient(hunting): get_auth_request con 0x7fc460005430 auth_method 0 2024-09-16T07:20:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:49.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc4797d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:49.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc4797d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:49.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc4797d4700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:20:49.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:49.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:49.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:20:49.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient(hunting): handle_auth_done global_id 4966 payload 293 2024-09-16T07:20:49.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 -- 172.21.15.37:0/3672900931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc460005430 msgr2=0x7fc46000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:49.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc460005430 0x7fc46000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:49.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 -- 172.21.15.37:0/3672900931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc460003a20 msgr2=0x7fc460003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:49.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc460003a20 0x7fc460003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:49.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient: _finish_hunting 0 2024-09-16T07:20:49.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 monclient: found mon.c 2024-09-16T07:20:49.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:49.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc479fd5700 1 -- 172.21.15.37:0/3672900931 reap_dead start 2024-09-16T07:20:49.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:49.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 1 -- 172.21.15.37:0/3672900931 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4600035d0 con 0x7fc460004870 2024-09-16T07:20:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient: _finish_auth 0 2024-09-16T07:20:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 10 monclient: _check_auth_tickets 2024-09-16T07:20:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.808+0000 7fc478fd3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc478fd3700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc460004cf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc464009860 tx=0x7fc4600058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f757ef9d7f1563f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464004230 con 0x7fc460004870 2024-09-16T07:20:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc46bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc46bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc46bfff700 10 monclient: dump: 2024-09-16T07:20:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:49.828 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:49.828 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:49.828 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:49.828 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:49.828 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:49.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:49.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4640043b0 con 0x7fc460004870 2024-09-16T07:20:49.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc46bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:49.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:49.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 4 set_mon_vals no callback set 2024-09-16T07:20:49.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc46bfff700 1 -- 172.21.15.37:0/3672900931 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4640036c0 con 0x7fc460004870 2024-09-16T07:20:49.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc46bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:49.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc46bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:49.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc46bfff700 10 monclient: dump: 2024-09-16T07:20:49.830 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:49.830 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:49.830 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:49.830 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:49.831 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:49.831 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:49.831 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:49.831 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:49.831 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:49.831 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:49.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 10 monclient: shutdown 2024-09-16T07:20:49.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:49.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3672900931 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 msgr2=0x7fc460004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:49.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc460004cf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc464009860 tx=0x7fc4600058a0 comp rx=0 tx=0).stop 2024-09-16T07:20:49.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3672900931 shutdown_connections 2024-09-16T07:20:49.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 1 --2- 172.21.15.37:0/3672900931 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc460004cf0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:49.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3672900931 >> 172.21.15.37:0/3672900931 conn(0x7fc4740a14c0 msgr2=0x7fc4740a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:49.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3672900931 shutdown_connections 2024-09-16T07:20:49.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3672900931 wait complete. 2024-09-16T07:20:49.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 10 monclient: build_initial_monmap 2024-09-16T07:20:49.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.812+0000 7fc47a7d6700 10 monclient: monmap: 2024-09-16T07:20:49.833 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:49.834 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:49.834 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:49.834 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:49.834 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:49.834 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:49.834 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:49.835 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:49.835 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:49.835 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:49.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 1 Processor -- start 2024-09-16T07:20:49.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 1 -- start start 2024-09-16T07:20:49.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient: init 2024-09-16T07:20:49.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:49.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:20:49.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc474132570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:49.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient(hunting): picked mon.noname-c con 0x7fc460004870 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:49.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4741c5be0 0x7fc474132ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:49.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient(hunting): picked mon.noname-b con 0x7fc4741c5be0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:49.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc47411bff0 0x7fc47411c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:49.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient(hunting): picked mon.noname-a con 0x7fc47411bff0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:49.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:49.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47411f6b0 con 0x7fc47411bff0 2024-09-16T07:20:49.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:49.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4741c68b0 con 0x7fc4741c5be0 2024-09-16T07:20:49.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:49.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc474072660 con 0x7fc460004870 2024-09-16T07:20:49.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:49.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient(hunting): authenticate will time out at 970.305190s 2024-09-16T07:20:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc474132570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc479fd5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc47411bff0 0x7fc47411c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc474132570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:36998/0 (socket says 172.21.15.37:36998) 2024-09-16T07:20:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 1 -- 172.21.15.37:0/3919287737 learned_addr learned my addr 172.21.15.37:0/3919287737 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc479fd5700 10 monclient(hunting): get_auth_request con 0x7fc47411bff0 auth_method 0 2024-09-16T07:20:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc479fd5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:49.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc479fd5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:49.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc479fd5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:49.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 10 monclient(hunting): get_auth_request con 0x7fc460004870 auth_method 0 2024-09-16T07:20:49.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:49.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:49.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:49.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc478fd3700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4741c5be0 0x7fc474132ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:49.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc478fd3700 10 monclient(hunting): get_auth_request con 0x7fc4741c5be0 auth_method 0 2024-09-16T07:20:49.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc478fd3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:49.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc478fd3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:49.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc478fd3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:49.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc479fd5700 1 -- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc47411bff0 msgr2=0x7fc47411c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:20:49.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc479fd5700 1 -- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc47411bff0 msgr2=0x7fc47411c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:20:49.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc479fd5700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc47411bff0 0x7fc47411c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:20:49.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:49.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:49.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc479fd5700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc47411bff0 0x7fc47411c470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:20:49.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:49.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc478fd3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:49.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc478fd3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:49.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc478fd3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:49.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 10 monclient(hunting): handle_auth_done global_id 4778 payload 824 2024-09-16T07:20:49.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 1 -- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4741c5be0 msgr2=0x7fc474132ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:49.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4741c5be0 0x7fc474132ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:49.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 1 -- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc47411bff0 msgr2=0x7fc47411c470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:20:49.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc47411bff0 0x7fc47411c470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:49.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 10 monclient: _finish_hunting 0 2024-09-16T07:20:49.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 1 monclient: found mon.noname-c 2024-09-16T07:20:49.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:49.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:20:49.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 1 -- 172.21.15.37:0/3919287737 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47411ff50 con 0x7fc460004870 2024-09-16T07:20:49.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc478fd3700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4741c5be0 0x7fc474132ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:20:49.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc4797d4700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc474132570 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc47000f910 tx=0x7fc47000fc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:49.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 1 -- 172.21.15.37:0/3919287737 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47000fe60 con 0x7fc460004870 2024-09-16T07:20:49.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:49.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:20:49.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 10 monclient: dump: 2024-09-16T07:20:49.853 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:49.853 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:49.853 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:49.853 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:49.853 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:49.853 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:49.854 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:49.854 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:49.854 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:49.854 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:49.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 10 monclient: _finish_auth 0 2024-09-16T07:20:49.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:20:49.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:49.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 1 -- 172.21.15.37:0/3919287737 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4700052e0 con 0x7fc460004870 2024-09-16T07:20:49.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 5 monclient: authenticate success, global_id 4778 2024-09-16T07:20:49.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient: _renew_subs 2024-09-16T07:20:49.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:49.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3919287737 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47409db90 con 0x7fc460004870 2024-09-16T07:20:49.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:49.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 1 -- 172.21.15.37:0/3919287737 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4700036a0 con 0x7fc460004870 2024-09-16T07:20:49.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:49.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:49.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc469ffb700 10 monclient: dump: 2024-09-16T07:20:49.856 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:49.856 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:49.856 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:49.856 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:49.856 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:49.856 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:49.856 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc46b7fe700 4 set_mon_vals no callback set 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient: _renew_subs 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.816+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3919287737 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47411fc70 con 0x7fc460004870 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.820+0000 7fc47a7d6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.820+0000 7fc47a7d6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.820+0000 7fc47a7d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.820+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3919287737 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47411f7f0 con 0x7fc460004870 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.820+0000 7fc469ffb700 1 -- 172.21.15.37:0/3919287737 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fc470003800 con 0x7fc460004870 2024-09-16T07:20:49.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.820+0000 7fc469ffb700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc458035630 0x7fc458037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:49.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.820+0000 7fc469ffb700 1 -- 172.21.15.37:0/3919287737 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fc470047680 con 0x7fc460004870 2024-09-16T07:20:49.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.820+0000 7fc478fd3700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc458035630 0x7fc458037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:49.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.820+0000 7fc478fd3700 10 monclient: get_auth_request con 0x7fc458035630 auth_method 0 2024-09-16T07:20:49.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.820+0000 7fc478fd3700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc458035630 0x7fc458037af0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc464015040 tx=0x7fc464020040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:49.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.824+0000 7fc469ffb700 1 -- 172.21.15.37:0/3919287737 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc47411f7f0 con 0x7fc460004870 2024-09-16T07:20:49.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.824+0000 7fc469ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:49.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:49.824+0000 7fc469ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:50.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.056+0000 7fc47a7d6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:20:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.056+0000 7fc47a7d6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:20:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.056+0000 7fc47a7d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.056+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3919287737 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc47411f7f0 con 0x7fc460004870 2024-09-16T07:20:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.056+0000 7fc469ffb700 1 -- 172.21.15.37:0/3919287737 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7fc47411f7f0 con 0x7fc460004870 2024-09-16T07:20:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.056+0000 7fc469ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:20:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.056+0000 7fc469ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:50.060 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:20:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc458035630 msgr2=0x7fc458037af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc458035630 0x7fc458037af0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc464015040 tx=0x7fc464020040 comp rx=0 tx=0).stop 2024-09-16T07:20:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 10 monclient: shutdown 2024-09-16T07:20:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 msgr2=0x7fc474132570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc474132570 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc47000f910 tx=0x7fc47000fc20 comp rx=0 tx=0).stop 2024-09-16T07:20:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3919287737 shutdown_connections 2024-09-16T07:20:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc47411bff0 0x7fc47411c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4741c5be0 0x7fc474132ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fc458035630 0x7fc458037af0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 --2- 172.21.15.37:0/3919287737 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc460004870 0x7fc474132570 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3919287737 >> 172.21.15.37:0/3919287737 conn(0x7fc4740a14c0 msgr2=0x7fc4740a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3919287737 shutdown_connections 2024-09-16T07:20:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.060+0000 7fc47a7d6700 1 -- 172.21.15.37:0/3919287737 wait complete. 2024-09-16T07:20:50.074 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T07:20:50.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T07:20:50.075 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T07:20:50.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient: get_monmap_and_config 2024-09-16T07:20:50.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient: build_initial_monmap 2024-09-16T07:20:50.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient: monmap: 2024-09-16T07:20:50.187 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:50.187 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:50.187 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:50.188593+0000 2024-09-16T07:20:50.187 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:50.188593+0000 2024-09-16T07:20:50.187 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:50.187 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:50.187 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:50.188 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:50.188 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:50.188 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:50.188 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:50.188 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:50.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 1 Processor -- start 2024-09-16T07:20:50.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 1 -- start start 2024-09-16T07:20:50.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient: init 2024-09-16T07:20:50.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:50.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T07:20:50.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a2750 0x7fb4ac0a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:50.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient(hunting): picked mon.noname-b con 0x7fb4ac0a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:50.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4ac0a3070 addr v1:172.21.15.143:6789/0 2024-09-16T07:20:50.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb4ac133730 0x7fb4ac13b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:50.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient(hunting): picked mon.noname-a con 0x7fb4ac133730 addr v2:172.21.15.37:3300/0 2024-09-16T07:20:50.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ac099be0 con 0x7fb4ac0a3070 2024-09-16T07:20:50.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:50.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4ac09de40 con 0x7fb4ac133730 2024-09-16T07:20:50.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:50.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac09d530 con 0x7fb4ac0a2750 2024-09-16T07:20:50.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:50.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4b08ef700 10 monclient(hunting): authenticate will time out at 970.674539s 2024-09-16T07:20:50.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a2750 0x7fb4ac0a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:50.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4abfff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb4ac133730 0x7fb4ac13b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:50.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a2750 0x7fb4ac0a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37012/0 (socket says 172.21.15.37:37012) 2024-09-16T07:20:50.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/752246553 learned_addr learned my addr 172.21.15.37:0/752246553 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:50.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request con 0x7fb4ac0a2750 auth_method 0 2024-09-16T07:20:50.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:50.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:50.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:50.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4abfff700 10 monclient(hunting): get_auth_request con 0x7fb4ac133730 auth_method 0 2024-09-16T07:20:50.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:50.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4abfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:50.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4abfff700 1 -- 172.21.15.37:0/752246553 >> v2:172.21.15.37:3300/0 conn(0x7fb4ac133730 msgr2=0x7fb4ac13b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4abfff700 1 -- 172.21.15.37:0/752246553 >> v2:172.21.15.37:3300/0 conn(0x7fb4ac133730 msgr2=0x7fb4ac13b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4abfff700 1 --2- 172.21.15.37:0/752246553 >> v2:172.21.15.37:3300/0 conn(0x7fb4ac133730 0x7fb4ac13b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4abfff700 1 --2- 172.21.15.37:0/752246553 >> v2:172.21.15.37:3300/0 conn(0x7fb4ac133730 0x7fb4ac13b310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_done global_id 4781 payload 274 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/752246553 >> v2:172.21.15.37:3300/0 conn(0x7fb4ac133730 msgr2=0x7fb4ac13b310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/752246553 >> v2:172.21.15.37:3300/0 conn(0x7fb4ac133730 0x7fb4ac13b310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/752246553 >> v1:172.21.15.143:6789/0 conn(0x7fb4ac0a3070 legacy=0x7fb4ac133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:20:50.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 1 monclient: found mon.noname-b 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/752246553 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac09a020 con 0x7fb4ac0a2750 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.184+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/752246553 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a2750 0x7fb4ac0a2b30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb494009910 tx=0x7fb494009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f735261e27e27007 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 1 -- 172.21.15.37:0/752246553 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49400ba60 con 0x7fb4ac0a2750 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 10 monclient: dump: 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:50.198 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 10 monclient: _finish_auth 0 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 1 -- 172.21.15.37:0/752246553 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49400bbc0 con 0x7fb4ac0a2750 2024-09-16T07:20:50.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 5 monclient: authenticate success, global_id 4781 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 1 -- 172.21.15.37:0/752246553 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494009e60 con 0x7fb4ac0a2750 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 4 set_mon_vals no callback set 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4aa7fc700 10 monclient: dump: 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:50.200 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:50.201 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:50.201 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:50.201 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:50.201 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:50.201 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:50.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 10 monclient: shutdown 2024-09-16T07:20:50.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:50.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 1 -- 172.21.15.37:0/752246553 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a2750 msgr2=0x7fb4ac0a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:50.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 1 --2- 172.21.15.37:0/752246553 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a2750 0x7fb4ac0a2b30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb494009910 tx=0x7fb494009c20 comp rx=0 tx=0).stop 2024-09-16T07:20:50.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 1 -- 172.21.15.37:0/752246553 shutdown_connections 2024-09-16T07:20:50.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 1 --2- 172.21.15.37:0/752246553 >> v2:172.21.15.37:3300/0 conn(0x7fb4ac133730 0x7fb4ac13b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:50.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 1 --2- 172.21.15.37:0/752246553 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a2750 0x7fb4ac0a2b30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:50.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 1 -- 172.21.15.37:0/752246553 >> 172.21.15.37:0/752246553 conn(0x7fb4ac0a3ea0 msgr2=0x7fb4ac0a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:50.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 1 -- 172.21.15.37:0/752246553 shutdown_connections 2024-09-16T07:20:50.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 1 -- 172.21.15.37:0/752246553 wait complete. 2024-09-16T07:20:50.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 10 monclient: build_initial_monmap 2024-09-16T07:20:50.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 10 monclient: monmap: 2024-09-16T07:20:50.203 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:50.203 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:50.204 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:50.204 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:50.204 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:50.204 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:50.204 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:50.204 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:50.204 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:50.204 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:50.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.188+0000 7fb4b08ef700 1 Processor -- start 2024-09-16T07:20:50.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 1 -- start start 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient: init 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ac0a2750 0x7fb4ac1c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient(hunting): picked mon.noname-b con 0x7fb4ac0a2750 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a3070 0x7fb4ac132510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient(hunting): picked mon.noname-c con 0x7fb4ac0a3070 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac133730 0x7fb4ac132a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient(hunting): picked mon.noname-a con 0x7fb4ac133730 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:50.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac1c8630 con 0x7fb4ac133730 2024-09-16T07:20:50.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:50.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac1330b0 con 0x7fb4ac0a2750 2024-09-16T07:20:50.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:50.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac11e6d0 con 0x7fb4ac0a3070 2024-09-16T07:20:50.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:50.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient(hunting): authenticate will time out at 970.681301s 2024-09-16T07:20:50.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a3070 0x7fb4ac132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:50.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4abfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac133730 0x7fb4ac132a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:50.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a3070 0x7fb4ac132510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37024/0 (socket says 172.21.15.37:37024) 2024-09-16T07:20:50.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 1 -- 172.21.15.37:0/1011880622 learned_addr learned my addr 172.21.15.37:0/1011880622 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:50.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 10 monclient(hunting): get_auth_request con 0x7fb4ac0a3070 auth_method 0 2024-09-16T07:20:50.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:50.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:50.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:50.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4abfff700 10 monclient(hunting): get_auth_request con 0x7fb4ac133730 auth_method 0 2024-09-16T07:20:50.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4abfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4abfff700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac133730 msgr2=0x7fb4ac132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4abfff700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac133730 msgr2=0x7fb4ac132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4abfff700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac133730 0x7fb4ac132a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ac0a2750 0x7fb4ac1c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request con 0x7fb4ac0a2750 auth_method 0 2024-09-16T07:20:50.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4abfff700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac133730 0x7fb4ac132a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 10 monclient(hunting): handle_auth_done global_id 4784 payload 824 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ac0a2750 msgr2=0x7fb4ac1c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ac0a2750 0x7fb4ac1c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac133730 msgr2=0x7fb4ac132a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac133730 0x7fb4ac132a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:50.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 10 monclient: _finish_hunting 0 2024-09-16T07:20:50.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 1 monclient: found mon.noname-c 2024-09-16T07:20:50.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:50.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:20:50.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 1 -- 172.21.15.37:0/1011880622 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb494009570 con 0x7fb4ac0a3070 2024-09-16T07:20:50.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4aaffd700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a3070 0x7fb4ac132510 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00c500 tx=0x7fb49c00c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:50.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c0052d0 con 0x7fb4ac0a3070 2024-09-16T07:20:50.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:50.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:20:50.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 10 monclient: dump: 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 10 monclient: _finish_auth 0 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 10 monclient: _check_auth_tickets 2024-09-16T07:20:50.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 5 monclient: authenticate success, global_id 4784 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49c005430 con 0x7fb4ac0a3070 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient: _renew_subs 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 1 -- 172.21.15.37:0/1011880622 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ac11f910 con 0x7fb4ac0a3070 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c0055a0 con 0x7fb4ac0a3070 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a3fff700 10 monclient: dump: 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:50.217 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4a9ffb700 4 set_mon_vals no callback set 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient: _renew_subs 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 1 -- 172.21.15.37:0/1011880622 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ac11e420 con 0x7fb4ac0a3070 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.192+0000 7fb4b08ef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:50.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4b08ef700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4b08ef700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4b08ef700 1 -- 172.21.15.37:0/1011880622 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac067530 con 0x7fb4ac11bf50 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4ab7fe700 10 monclient: get_auth_request con 0x7fb4ac11bf50 auth_method 0 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 msgr2=0x7fb4ac11c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 msgr2=0x7fb4ac11c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fb49c0038e0 con 0x7fb4ac0a3070 2024-09-16T07:20:50.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:20:50.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4a3fff700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb490035600 0x7fb490037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:50.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fb49c046e40 con 0x7fb4ac0a3070 2024-09-16T07:20:50.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4abfff700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb490035600 0x7fb490037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:50.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4abfff700 10 monclient: get_auth_request con 0x7fb490035600 auth_method 0 2024-09-16T07:20:50.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.196+0000 7fb4abfff700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb490035600 0x7fb490037ac0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009610 tx=0x7fb4a400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.400+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.400+0000 7fb4ab7fe700 10 monclient: get_auth_request con 0x7fb4ac11bf50 auth_method 0 2024-09-16T07:20:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.400+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.400+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:50.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.400+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:50.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.400+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 msgr2=0x7fb4ac11c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:20:50.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.400+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 msgr2=0x7fb4ac11c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:20:50.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.400+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:20:50.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.400+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:20:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.804+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.804+0000 7fb4ab7fe700 10 monclient: get_auth_request con 0x7fb4ac11bf50 auth_method 0 2024-09-16T07:20:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.804+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.804+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.804+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.804+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 msgr2=0x7fb4ac11c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:20:50.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.804+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 msgr2=0x7fb4ac11c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:20:50.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.804+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:20:50.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:50.804+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:20:51.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:51.608+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:51.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:51.608+0000 7fb4ab7fe700 10 monclient: get_auth_request con 0x7fb4ac11bf50 auth_method 0 2024-09-16T07:20:51.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:51.608+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:51.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:51.608+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:51.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:51.608+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:51.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:51.608+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 msgr2=0x7fb4ac11c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:20:51.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:51.608+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 msgr2=0x7fb4ac11c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:20:51.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:51.608+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:20:51.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:51.608+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T07:20:51.751 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:51.751 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25628]: starting. 2024-09-16T07:20:51.751 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25628]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25591 2024-09-16T07:20:51.751 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25628]: created object 0... 2024-09-16T07:20:51.751 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25628]: created object 25... 2024-09-16T07:20:51.751 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25628]: created object 49... 2024-09-16T07:20:51.751 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25628]: finishing. 2024-09-16T07:20:51.751 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25628]: shutting down. 2024-09-16T07:20:51.781 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:51.781 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25630]: starting. 2024-09-16T07:20:51.781 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25630]: listing objects. 2024-09-16T07:20:51.781 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25630]: listed object 0... 2024-09-16T07:20:51.781 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25630]: listed object 25... 2024-09-16T07:20:51.781 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25630]: saw 50 objects 2024-09-16T07:20:51.781 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25630]: shutting down. 2024-09-16T07:20:52.622 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:52.622 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25629]: starting. 2024-09-16T07:20:52.622 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25629]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25591 2024-09-16T07:20:52.622 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25629]: shutting down. 2024-09-16T07:20:52.624 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:52.624 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:20:52.624 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:20:52.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:20:53.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.192+0000 7fb4a2ffd700 10 monclient: tick 2024-09-16T07:20:53.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.192+0000 7fb4a2ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:20:53.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.192+0000 7fb4a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:53.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.192+0000 7fb4a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 10 monclient: get_auth_request con 0x7fb4ac11bf50 auth_method 0 2024-09-16T07:20:53.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:53.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_done global_id 34104 payload 824 2024-09-16T07:20:53.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4ac06dbf0 con 0x7fb4ac11bf50 2024-09-16T07:20:53.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb494009910 tx=0x7fb494004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49401b8c0 con 0x7fb4ac11bf50 2024-09-16T07:20:53.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:53.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 10 monclient: dump: 2024-09-16T07:20:53.216 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.216 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.216 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.216 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.216 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.217 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.217 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.217 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.217 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.217 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fb49401ba20 con 0x7fb4ac11bf50 2024-09-16T07:20:53.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:53.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:53.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 msgr2=0x7fb4ac11c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb494009910 tx=0x7fb494004770 comp rx=0 tx=0).stop 2024-09-16T07:20:53.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 msgr2=0x7fb4ac11c330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:53.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.212+0000 7fb4a3fff700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11bf50 0x7fb4ac11c330 secure :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb494009910 tx=0x7fb494004770 comp rx=0 tx=0).stop 2024-09-16T07:20:53.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4b08ef700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4b08ef700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 0x7fb4ac06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4b08ef700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4b08ef700 1 -- 172.21.15.37:0/1011880622 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac0a3450 con 0x7fb4ac11c870 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 0x7fb4ac06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4ab7fe700 10 monclient: get_auth_request con 0x7fb4ac11c870 auth_method 0 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 msgr2=0x7fb4ac06e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 msgr2=0x7fb4ac06e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 0x7fb4ac06e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:20:53.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.228+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 0x7fb4ac06e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:20:53.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 0x7fb4ac06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 10 monclient: get_auth_request con 0x7fb4ac11c870 auth_method 0 2024-09-16T07:20:53.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:53.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 10 monclient(hunting): handle_auth_done global_id 34110 payload 824 2024-09-16T07:20:53.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 1 -- 172.21.15.37:0/1011880622 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb4ac0724c0 con 0x7fb4ac11c870 2024-09-16T07:20:53.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4ab7fe700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 0x7fb4ac06e320 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb494009c60 tx=0x7fb4940092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494027a70 con 0x7fb4ac11c870 2024-09-16T07:20:53.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:53.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 10 monclient: dump: 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4453 (secure 0 0 0) 0x7fb494027bd0 con 0x7fb4ac11c870 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:20:53.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:53.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 msgr2=0x7fb4ac06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 0x7fb4ac06e320 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb494009c60 tx=0x7fb4940092e0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 msgr2=0x7fb4ac06e320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:53.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4a3fff700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac11c870 0x7fb4ac06e320 secure :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb494009c60 tx=0x7fb4940092e0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.438 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:20:53.438 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:20:53.438 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:20:53.438 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:20:53.438 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 16, 2024-09-16T07:20:53.438 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:53.439 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:53.440 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:20:53.441 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:53.442 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.443 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.444 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:53.445 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:20:53.446 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.447 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:53.448 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.449 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:53.450 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:53.450 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:20:53.450 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.450 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.450 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.450 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:20:53.450 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:20:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4b08ef700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb490035600 msgr2=0x7fb490037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4b08ef700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb490035600 0x7fb490037ac0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009610 tx=0x7fb4a400b040 comp rx=0 tx=0).stop 2024-09-16T07:20:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4b08ef700 10 monclient: shutdown 2024-09-16T07:20:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4b08ef700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4b08ef700 1 -- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a3070 msgr2=0x7fb4ac132510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4b08ef700 1 --2- 172.21.15.37:0/1011880622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4ac0a3070 0x7fb4ac132510 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb49c00c500 tx=0x7fb49c00c810 comp rx=0 tx=0).stop 2024-09-16T07:20:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4abfff700 1 -- 172.21.15.37:0/1011880622 reap_dead start 2024-09-16T07:20:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4abfff700 1 -- 172.21.15.37:0/1011880622 reap_dead start 2024-09-16T07:20:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4b08ef700 1 -- 172.21.15.37:0/1011880622 shutdown_connections 2024-09-16T07:20:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.432+0000 7fb4b08ef700 1 -- 172.21.15.37:0/1011880622 >> 172.21.15.37:0/1011880622 conn(0x7fb4ac0a3ea0 msgr2=0x7fb4ac06d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.436+0000 7fb4b08ef700 1 -- 172.21.15.37:0/1011880622 shutdown_connections 2024-09-16T07:20:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.436+0000 7fb4b08ef700 1 -- 172.21.15.37:0/1011880622 wait complete. 2024-09-16T07:20:53.453 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:20:53.453 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T07:20:53.453 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T07:20:53.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient: get_monmap_and_config 2024-09-16T07:20:53.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient: build_initial_monmap 2024-09-16T07:20:53.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient: monmap: 2024-09-16T07:20:53.578 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:53.578 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.578 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:53.579598+0000 2024-09-16T07:20:53.578 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:53.579598+0000 2024-09-16T07:20:53.578 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:53.578 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.578 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:53.579 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:53.579 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:53.579 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:53.579 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:53.579 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 Processor -- start 2024-09-16T07:20:53.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 -- start start 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient: init 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 0x7f72440a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient(hunting): picked mon.noname-b con 0x7f72440a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f72440a1270 0x7f72440a2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient(hunting): picked mon.noname-a con 0x7f72440a1270 addr v2:172.21.15.37:3300/0 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72440a2b80 addr v1:172.21.15.37:6789/0 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72440a36c0 con 0x7f72440a2b80 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f724406cf50 con 0x7f72440a1270 2024-09-16T07:20:53.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724409f810 con 0x7f72440a0e90 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient(hunting): authenticate will time out at 974.065826s 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724918c700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f72440a2b80 0x7f72440a2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50912/0 (socket says 172.21.15.37:50912) 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724918c700 1 -- 172.21.15.37:0/3875316620 learned_addr learned my addr 172.21.15.37:0/3875316620 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f7243fff700 1 --2- 172.21.15.37:0/3875316620 >> v2:172.21.15.37:3300/0 conn(0x7f72440a1270 0x7f72440a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 1 --2- 172.21.15.37:0/3875316620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 0x7f72440a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 10 monclient(hunting): get_auth_request con 0x7f72440a0e90 auth_method 0 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f7243fff700 10 monclient(hunting): get_auth_request con 0x7f72440a1270 auth_method 0 2024-09-16T07:20:53.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f7243fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f7243fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f7243fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f7243fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f7243fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f7243fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 1 -- 172.21.15.37:0/3875316620 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067890168 0 0) 0x7f72440a36c0 con 0x7f72440a2b80 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient(hunting): _negotiate 2024-09-16T07:20:53.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient(hunting): my global_id is 34116 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 1 -- 172.21.15.37:0/3875316620 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f722c0035d0 con 0x7f72440a2b80 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 10 monclient(hunting): handle_auth_done global_id 4799 payload 274 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 1 -- 172.21.15.37:0/3875316620 >> v2:172.21.15.37:3300/0 conn(0x7f72440a1270 msgr2=0x7f72440a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 1 --2- 172.21.15.37:0/3875316620 >> v2:172.21.15.37:3300/0 conn(0x7f72440a1270 0x7f72440a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 1 -- 172.21.15.37:0/3875316620 >> v1:172.21.15.37:6789/0 conn(0x7f72440a2b80 legacy=0x7f72440a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 10 monclient: _finish_hunting 0 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 1 monclient: found mon.noname-b 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:20:53.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 1 -- 172.21.15.37:0/3875316620 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724409f250 con 0x7f72440a0e90 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f7243fff700 1 --2- 172.21.15.37:0/3875316620 >> v2:172.21.15.37:3300/0 conn(0x7f72440a1270 0x7f72440a2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724898b700 1 --2- 172.21.15.37:0/3875316620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 0x7f72440a04d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7238009850 tx=0x7f7238009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6ac68a547665d08 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 1 -- 172.21.15.37:0/3875316620 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238011830 con 0x7f72440a0e90 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient: dump: 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.587 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient: _finish_auth 0 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient: _check_auth_tickets 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 5 monclient: authenticate success, global_id 4799 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 1 -- 172.21.15.37:0/3875316620 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7238011990 con 0x7f72440a0e90 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 4 set_mon_vals no callback set 2024-09-16T07:20:53.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 1 -- 172.21.15.37:0/3875316620 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238011d40 con 0x7f72440a0e90 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f72437fe700 10 monclient: dump: 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 10 monclient: shutdown 2024-09-16T07:20:53.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:53.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 -- 172.21.15.37:0/3875316620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 msgr2=0x7f72440a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 --2- 172.21.15.37:0/3875316620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 0x7f72440a04d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7238009850 tx=0x7f7238009b60 comp rx=0 tx=0).stop 2024-09-16T07:20:53.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 -- 172.21.15.37:0/3875316620 shutdown_connections 2024-09-16T07:20:53.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 --2- 172.21.15.37:0/3875316620 >> v2:172.21.15.37:3300/0 conn(0x7f72440a1270 0x7f72440a2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 --2- 172.21.15.37:0/3875316620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 0x7f72440a04d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.576+0000 7f724998d700 1 -- 172.21.15.37:0/3875316620 >> 172.21.15.37:0/3875316620 conn(0x7f72440a4740 msgr2=0x7f72440a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 -- 172.21.15.37:0/3875316620 shutdown_connections 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 -- 172.21.15.37:0/3875316620 wait complete. 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient: build_initial_monmap 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient: monmap: 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:53.594 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:53.595 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:53.595 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:53.595 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 Processor -- start 2024-09-16T07:20:53.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 -- start start 2024-09-16T07:20:53.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient: init 2024-09-16T07:20:53.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:53.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:20:53.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 0x7f724413bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient(hunting): picked mon.noname-c con 0x7f72440a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:53.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72440a1270 0x7f724413c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient(hunting): picked mon.noname-b con 0x7f72440a1270 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:53.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72440a2b80 0x7f7244122c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient(hunting): picked mon.noname-a con 0x7f72440a2b80 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:53.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724413c950 con 0x7f72440a2b80 2024-09-16T07:20:53.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7244126890 con 0x7f72440a1270 2024-09-16T07:20:53.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7244126b70 con 0x7f72440a0e90 2024-09-16T07:20:53.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:53.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient(hunting): authenticate will time out at 974.070288s 2024-09-16T07:20:53.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 0x7f724413bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 0x7f724413bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38070/0 (socket says 172.21.15.37:38070) 2024-09-16T07:20:53.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 1 -- 172.21.15.37:0/2519389157 learned_addr learned my addr 172.21.15.37:0/2519389157 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:53.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 10 monclient(hunting): get_auth_request con 0x7f72440a0e90 auth_method 0 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724918c700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72440a2b80 0x7f7244122c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724918c700 10 monclient(hunting): get_auth_request con 0x7f72440a2b80 auth_method 0 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724918c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724918c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724918c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724918c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724918c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724918c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 10 monclient(hunting): handle_auth_done global_id 4802 payload 824 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 1 -- 172.21.15.37:0/2519389157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72440a1270 msgr2=0x7f724413c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72440a1270 0x7f724413c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 1 -- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72440a2b80 msgr2=0x7f7244122c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72440a2b80 0x7f7244122c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 10 monclient: _finish_hunting 0 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 1 monclient: found mon.noname-c 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 1 -- 172.21.15.37:0/2519389157 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72340094b0 con 0x7f72440a0e90 2024-09-16T07:20:53.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724898b700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 0x7f724413bb20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f72380090b0 tx=0x7f7238004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238004590 con 0x7f72440a0e90 2024-09-16T07:20:53.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:20:53.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 10 monclient: dump: 2024-09-16T07:20:53.608 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.608 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.608 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.608 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.608 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.608 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 10 monclient: _finish_auth 0 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 10 monclient: _check_auth_tickets 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 5 monclient: authenticate success, global_id 4802 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient: _renew_subs 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723801f070 con 0x7f72440a0e90 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:53.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 -- 172.21.15.37:0/2519389157 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72380094b0 con 0x7f72440a0e90 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7238003900 con 0x7f72440a0e90 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f7242ffd700 4 set_mon_vals no callback set 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f72417fa700 10 monclient: dump: 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.610 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.611 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.611 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.611 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.611 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient: _renew_subs 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.580+0000 7f724998d700 1 -- 172.21.15.37:0/2519389157 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72440a74f0 con 0x7f72440a0e90 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f724998d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f724998d700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123190 0x7f7244123570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f724998d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f724998d700 1 -- 172.21.15.37:0/2519389157 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7244067400 con 0x7f7244123190 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123190 0x7f7244123570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 10 monclient: get_auth_request con 0x7f7244123190 auth_method 0 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 10 monclient(hunting): handle_auth_done global_id 4805 payload 824 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 1 -- 172.21.15.37:0/2519389157 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72440a3f00 con 0x7f7244123190 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f7238003a60 con 0x7f72440a0e90 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f7243fff700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123190 0x7f7244123570 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7244126aa0 tx=0x7f723400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7230035630 0x7f7230037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f7238021070 con 0x7f72440a0e90 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723400c720 con 0x7f7244123190 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:53.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 10 monclient: dump: 2024-09-16T07:20:53.614 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.614 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.614 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.614 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.614 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.614 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.614 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.615 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.615 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.615 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f723400c880 con 0x7f7244123190 2024-09-16T07:20:53.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:53.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:53.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f724918c700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7230035630 0x7f7230037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123190 msgr2=0x7f7244123570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123190 0x7f7244123570 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7244126aa0 tx=0x7f723400ee70 comp rx=0 tx=0).stop 2024-09-16T07:20:53.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123190 msgr2=0x7f7244123570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:53.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f72417fa700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123190 0x7f7244123570 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f724918c700 10 monclient: get_auth_request con 0x7f7230035630 auth_method 0 2024-09-16T07:20:53.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.584+0000 7f724918c700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7230035630 0x7f7230037af0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f724406d830 tx=0x7f723c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:20:53.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123ab0 0x7f724411f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 1 -- 172.21.15.37:0/2519389157 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72440a1670 con 0x7f7244123ab0 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123ab0 0x7f724411f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 10 monclient: get_auth_request con 0x7f7244123ab0 auth_method 0 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 10 monclient(hunting): handle_auth_done global_id 4808 payload 824 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 1 -- 172.21.15.37:0/2519389157 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f72440a54d0 con 0x7f7244123ab0 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f7243fff700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123ab0 0x7f724411f2a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7234019000 tx=0x7f7234009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234013670 con 0x7f7244123ab0 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 10 monclient: dump: 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.618 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.619 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.619 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4619 (secure 0 0 0) 0x7f7234003e20 con 0x7f7244123ab0 2024-09-16T07:20:53.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:20:53.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:53.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123ab0 msgr2=0x7f724411f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123ab0 0x7f724411f2a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7234019000 tx=0x7f7234009240 comp rx=0 tx=0).stop 2024-09-16T07:20:53.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 1 -- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123ab0 msgr2=0x7f724411f2a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:53.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f72417fa700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7244123ab0 0x7f724411f2a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.620 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:20:53.620 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:20:53.620 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:20:53.620 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:20:53.620 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 16, 2024-09-16T07:20:53.620 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:20:53.620 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:20:53.620 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:20:53.620 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:20:53.620 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:53.621 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:53.622 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:53.623 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:53.623 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:53.623 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:53.623 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:53.623 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.623 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:20:53.623 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:20:53.623 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:20:53.623 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:20:53.624 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:20:53.624 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:20:53.624 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:53.624 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:53.624 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:20:53.624 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:20:53.624 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:20:53.624 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:20:53.624 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:20:53.625 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:20:53.625 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:20:53.625 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:53.625 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:20:53.625 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:53.625 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:53.625 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:53.625 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:53.625 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:53.625 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:20:53.626 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.627 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:20:53.628 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:53.629 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:53.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.629 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:53.629 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:20:53.629 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.629 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:53.629 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:20:53.629 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.630 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:20:53.631 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:20:53.631 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:53.631 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:53.631 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.631 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:53.631 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:20:53.631 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.631 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.631 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.631 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:53.632 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:20:53.632 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.632 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.632 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.632 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:20:53.632 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:20:53.632 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:53.632 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:53.632 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.633 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:53.634 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:20:53.635 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.635 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.635 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:20:53.635 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:20:53.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 1 -- 172.21.15.37:0/2519389157 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7230035630 msgr2=0x7f7230037af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7230035630 0x7f7230037af0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f724406d830 tx=0x7f723c009040 comp rx=0 tx=0).stop 2024-09-16T07:20:53.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 10 monclient: shutdown 2024-09-16T07:20:53.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:53.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 1 -- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 msgr2=0x7f724413bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 1 --2- 172.21.15.37:0/2519389157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f72440a0e90 0x7f724413bb20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f72380090b0 tx=0x7f7238004770 comp rx=0 tx=0).stop 2024-09-16T07:20:53.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724918c700 1 -- 172.21.15.37:0/2519389157 reap_dead start 2024-09-16T07:20:53.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724918c700 1 -- 172.21.15.37:0/2519389157 reap_dead start 2024-09-16T07:20:53.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 1 -- 172.21.15.37:0/2519389157 shutdown_connections 2024-09-16T07:20:53.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 1 -- 172.21.15.37:0/2519389157 >> 172.21.15.37:0/2519389157 conn(0x7f72440a4740 msgr2=0x7f724409c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:53.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 1 -- 172.21.15.37:0/2519389157 shutdown_connections 2024-09-16T07:20:53.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.596+0000 7f724998d700 1 -- 172.21.15.37:0/2519389157 wait complete. 2024-09-16T07:20:53.638 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:20:53.638 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T07:20:53.638 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T07:20:53.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.760+0000 7fbe861f6700 10 monclient: get_monmap_and_config 2024-09-16T07:20:53.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.760+0000 7fbe861f6700 10 monclient: build_initial_monmap 2024-09-16T07:20:53.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.760+0000 7fbe861f6700 10 monclient: monmap: 2024-09-16T07:20:53.765 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:53.765 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.765 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:53.765959+0000 2024-09-16T07:20:53.765 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:53.765959+0000 2024-09-16T07:20:53.765 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:53.766 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.766 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:53.766 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:53.766 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:53.766 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:53.766 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:53.766 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.760+0000 7fbe861f6700 1 Processor -- start 2024-09-16T07:20:53.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.760+0000 7fbe861f6700 1 -- start start 2024-09-16T07:20:53.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.760+0000 7fbe861f6700 10 monclient: init 2024-09-16T07:20:53.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:53.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T07:20:53.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe80131a70 addr v1:172.21.15.37:6789/0 2024-09-16T07:20:53.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 0x7fbe80136960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 10 monclient(hunting): picked mon.noname-b con 0x7fbe8009c810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:53.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe80136ea0 addr v1:172.21.15.143:6789/0 2024-09-16T07:20:53.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8009b590 con 0x7fbe80131a70 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8009b870 con 0x7fbe80136ea0 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe851f4700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fbe80131a70 0x7fbe80131e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50920/0 (socket says 172.21.15.37:50920) 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe801325f0 con 0x7fbe8009c810 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe851f4700 1 -- 172.21.15.37:0/256292105 learned_addr learned my addr 172.21.15.37:0/256292105 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 10 monclient(hunting): authenticate will time out at 974.253316s 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 1 --2- 172.21.15.37:0/256292105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 0x7fbe80136960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 10 monclient(hunting): get_auth_request con 0x7fbe8009c810 auth_method 0 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 1 -- 172.21.15.37:0/256292105 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146190947 0 0) 0x7fbe8009b590 con 0x7fbe80131a70 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient(hunting): _negotiate 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient(hunting): my global_id is 34122 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 1 -- 172.21.15.37:0/256292105 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe6c0035d0 con 0x7fbe80131a70 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 10 monclient(hunting): handle_auth_done global_id 4811 payload 274 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 1 -- 172.21.15.37:0/256292105 >> v1:172.21.15.143:6789/0 conn(0x7fbe80136ea0 legacy=0x7fbe80139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 1 -- 172.21.15.37:0/256292105 >> v1:172.21.15.37:6789/0 conn(0x7fbe80131a70 legacy=0x7fbe80131e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 10 monclient: _finish_hunting 0 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 1 monclient: found mon.noname-b 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 1 -- 172.21.15.37:0/256292105 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8009bb50 con 0x7fbe8009c810 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe849f3700 1 --2- 172.21.15.37:0/256292105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 0x7fbe80136960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009880 tx=0x7fbe70009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38784ac4e109292b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 1 -- 172.21.15.37:0/256292105 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70011830 con 0x7fbe8009c810 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient: dump: 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.781 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient: _finish_auth 0 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient: _check_auth_tickets 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 5 monclient: authenticate success, global_id 4811 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 1 -- 172.21.15.37:0/256292105 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70011990 con 0x7fbe8009c810 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:53.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 4 set_mon_vals no callback set 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 1 -- 172.21.15.37:0/256292105 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70011d20 con 0x7fbe8009c810 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe77fff700 10 monclient: dump: 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.783 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 10 monclient: shutdown 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 -- 172.21.15.37:0/256292105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 msgr2=0x7fbe80136960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 --2- 172.21.15.37:0/256292105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 0x7fbe80136960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009880 tx=0x7fbe70009b90 comp rx=0 tx=0).stop 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 -- 172.21.15.37:0/256292105 shutdown_connections 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 --2- 172.21.15.37:0/256292105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 0x7fbe80136960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 -- 172.21.15.37:0/256292105 >> 172.21.15.37:0/256292105 conn(0x7fbe800a3f60 msgr2=0x7fbe800a4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 -- 172.21.15.37:0/256292105 shutdown_connections 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.764+0000 7fbe861f6700 1 -- 172.21.15.37:0/256292105 wait complete. 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient: build_initial_monmap 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient: monmap: 2024-09-16T07:20:53.784 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 1 Processor -- start 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 1 -- start start 2024-09-16T07:20:53.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient: init 2024-09-16T07:20:53.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:53.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:20:53.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 0x7fbe801ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient(hunting): picked mon.noname-c con 0x7fbe8009c810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:53.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe80131a70 0x7fbe801ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient(hunting): picked mon.noname-a con 0x7fbe80131a70 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:53.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe80136ea0 0x7fbe8011d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient(hunting): picked mon.noname-b con 0x7fbe80136ea0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:53.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe800a0460 con 0x7fbe80131a70 2024-09-16T07:20:53.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8011f730 con 0x7fbe80136ea0 2024-09-16T07:20:53.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8009b150 con 0x7fbe8009c810 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient(hunting): authenticate will time out at 974.258195s 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe851f4700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 0x7fbe801ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe80131a70 0x7fbe801ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe851f4700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 0x7fbe801ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38094/0 (socket says 172.21.15.37:38094) 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe851f4700 1 -- 172.21.15.37:0/3057430950 learned_addr learned my addr 172.21.15.37:0/3057430950 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe851f4700 10 monclient(hunting): get_auth_request con 0x7fbe8009c810 auth_method 0 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe851f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe851f4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe851f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 10 monclient(hunting): get_auth_request con 0x7fbe80131a70 auth_method 0 2024-09-16T07:20:53.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe859f5700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe80136ea0 0x7fbe8011d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe851f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe851f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe851f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe859f5700 10 monclient(hunting): get_auth_request con 0x7fbe80136ea0 auth_method 0 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe859f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe859f5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe859f5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 10 monclient(hunting): handle_auth_done global_id 34125 payload 824 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 1 -- 172.21.15.37:0/3057430950 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 msgr2=0x7fbe801ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe8009c810 0x7fbe801ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 1 -- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe80136ea0 msgr2=0x7fbe8011d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe80136ea0 0x7fbe8011d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 10 monclient: _finish_hunting 0 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 1 monclient: found mon.noname-a 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 1 -- 172.21.15.37:0/3057430950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe700094b0 con 0x7fbe80131a70 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe859f5700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe80136ea0 0x7fbe8011d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe849f3700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe80131a70 0x7fbe801ca820 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbe70015000 tx=0x7fbe70004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe700036c0 con 0x7fbe80131a70 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:53.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 10 monclient: dump: 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 10 monclient: _finish_auth 0 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:20:53.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:53.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 5 monclient: authenticate success, global_id 34125 2024-09-16T07:20:53.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70003820 con 0x7fbe80131a70 2024-09-16T07:20:53.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:53.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70003b90 con 0x7fbe80131a70 2024-09-16T07:20:53.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient: _renew_subs 2024-09-16T07:20:53.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:53.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 1 -- 172.21.15.37:0/3057430950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe800a00a0 con 0x7fbe80131a70 2024-09-16T07:20:53.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe777fe700 4 set_mon_vals no callback set 2024-09-16T07:20:53.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:53.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe75ffb700 10 monclient: dump: 2024-09-16T07:20:53.794 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.794 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.794 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.794 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.794 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.795 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.795 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.795 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.795 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.795 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient: _renew_subs 2024-09-16T07:20:53.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:53.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.768+0000 7fbe861f6700 1 -- 172.21.15.37:0/3057430950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe80120c50 con 0x7fbe80131a70 2024-09-16T07:20:53.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe861f6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:53.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fbe70003cf0 con 0x7fbe80131a70 2024-09-16T07:20:53.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe861f6700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011dd10 0x7fbe8002b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe861f6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe861f6700 1 -- 172.21.15.37:0/3057430950 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe80120de0 con 0x7fbe8011dd10 2024-09-16T07:20:53.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbe64035600 0x7fbe64037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 4738+0+0 (secure 0 0 0) 0x7fbe700490a0 con 0x7fbe80131a70 2024-09-16T07:20:53.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011dd10 0x7fbe8002b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 10 monclient: get_auth_request con 0x7fbe8011dd10 auth_method 0 2024-09-16T07:20:53.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe859f5700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbe64035600 0x7fbe64037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe859f5700 10 monclient: get_auth_request con 0x7fbe64035600 auth_method 0 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 10 monclient(hunting): handle_auth_done global_id 4996 payload 824 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 1 -- 172.21.15.37:0/3057430950 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe8009e410 con 0x7fbe8011dd10 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe851f4700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011dd10 0x7fbe8002b6a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c000f60 tx=0x7fbe7c00fb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe859f5700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbe64035600 0x7fbe64037ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe780012e0 tx=0x7fbe78009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c005b70 con 0x7fbe8011dd10 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 10 monclient: dump: 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fbe7c005cd0 con 0x7fbe8011dd10 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:53.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:53.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011dd10 msgr2=0x7fbe8002b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011dd10 0x7fbe8002b6a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c000f60 tx=0x7fbe7c00fb70 comp rx=0 tx=0).stop 2024-09-16T07:20:53.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011dd10 msgr2=0x7fbe8002b6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:53.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.772+0000 7fbe75ffb700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011dd10 0x7fbe8002b6a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe861f6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe861f6700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011e0f0 0x7fbe8002bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe861f6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe861f6700 1 -- 172.21.15.37:0/3057430950 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe80067700 con 0x7fbe8011e0f0 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011e0f0 0x7fbe8002bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 10 monclient: get_auth_request con 0x7fbe8011e0f0 auth_method 0 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 10 monclient(hunting): handle_auth_done global_id 4999 payload 824 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 1 -- 172.21.15.37:0/3057430950 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbe80135f00 con 0x7fbe8011e0f0 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe851f4700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011e0f0 0x7fbe8002bbe0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c000f60 tx=0x7fbe7c001150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c023070 con 0x7fbe8011e0f0 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe75ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.784+0000 7fbe75ffb700 10 monclient: dump: 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.802 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4787 (secure 0 0 0) 0x7fbe7c005e90 con 0x7fbe8011e0f0 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe75ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe75ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011e0f0 msgr2=0x7fbe8002bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe75ffb700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011e0f0 0x7fbe8002bbe0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c000f60 tx=0x7fbe7c001150 comp rx=0 tx=0).stop 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe75ffb700 1 -- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011e0f0 msgr2=0x7fbe8002bbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:53.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe75ffb700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe8011e0f0 0x7fbe8002bbe0 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c000f60 tx=0x7fbe7c001150 comp rx=0 tx=0).stop 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 16, 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.804 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:53.805 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:53.806 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:20:53.807 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:53.808 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:53.809 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:53.809 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:53.809 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.809 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:20:53.809 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.809 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:20:53.809 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:20:53.810 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:20:53.812 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:20:53.812 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:53.812 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:53.812 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.812 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:53.812 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:20:53.812 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.812 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:53.813 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.814 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:53.815 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:53.816 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.816 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.816 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.816 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:20:53.816 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:20:53.816 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.816 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:53.816 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:53.816 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.817 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:53.818 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:53.819 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:20:53.819 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:20:53.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe861f6700 1 -- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbe64035600 msgr2=0x7fbe64037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe861f6700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbe64035600 0x7fbe64037ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe780012e0 tx=0x7fbe78009040 comp rx=0 tx=0).stop 2024-09-16T07:20:53.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe861f6700 10 monclient: shutdown 2024-09-16T07:20:53.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe861f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:53.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe861f6700 1 -- 172.21.15.37:0/3057430950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe80131a70 msgr2=0x7fbe801ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe861f6700 1 --2- 172.21.15.37:0/3057430950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe80131a70 0x7fbe801ca820 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbe70015000 tx=0x7fbe70004770 comp rx=0 tx=0).stop 2024-09-16T07:20:53.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe859f5700 1 -- 172.21.15.37:0/3057430950 reap_dead start 2024-09-16T07:20:53.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe859f5700 1 -- 172.21.15.37:0/3057430950 reap_dead start 2024-09-16T07:20:53.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe861f6700 1 -- 172.21.15.37:0/3057430950 shutdown_connections 2024-09-16T07:20:53.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe861f6700 1 -- 172.21.15.37:0/3057430950 >> 172.21.15.37:0/3057430950 conn(0x7fbe800a3f60 msgr2=0x7fbe80071ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:53.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe861f6700 1 -- 172.21.15.37:0/3057430950 shutdown_connections 2024-09-16T07:20:53.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.788+0000 7fbe861f6700 1 -- 172.21.15.37:0/3057430950 wait complete. 2024-09-16T07:20:53.822 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:20:53.822 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T07:20:53.822 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T07:20:53.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.940+0000 7f2b22974700 10 monclient: get_monmap_and_config 2024-09-16T07:20:53.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.940+0000 7f2b22974700 10 monclient: build_initial_monmap 2024-09-16T07:20:53.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.940+0000 7f2b22974700 10 monclient: monmap: 2024-09-16T07:20:53.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:53.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.946 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:53.946394+0000 2024-09-16T07:20:53.946 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:53.946394+0000 2024-09-16T07:20:53.946 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:53.946 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.946 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:53.947 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:53.947 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:53.947 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:53.947 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:53.947 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.940+0000 7f2b22974700 1 Processor -- start 2024-09-16T07:20:53.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.940+0000 7f2b22974700 1 -- start start 2024-09-16T07:20:53.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.940+0000 7f2b22974700 10 monclient: init 2024-09-16T07:20:53.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:53.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-16T07:20:53.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b1c0a0e90 addr v1:172.21.15.37:6789/0 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b1c0a04d0 0x7f2b1c0a1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient(hunting): picked mon.noname-b con 0x7f2b1c0a04d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2b1c0a2d50 0x7f2b1c0a31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient(hunting): picked mon.noname-a con 0x7f2b1c0a2d50 addr v2:172.21.15.37:3300/0 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1c0a0900 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c0a17b0 con 0x7f2b1c0a2d50 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c06cf50 con 0x7f2b1c0a04d0 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:53.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient(hunting): authenticate will time out at 974.432334s 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b21171700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b1c0a04d0 0x7f2b1c0a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2b1c0a2d50 0x7f2b1c0a31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b21972700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2b1c0a0e90 0x7f2b1c0a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50926/0 (socket says 172.21.15.37:50926) 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b21972700 1 -- 172.21.15.37:0/3176932473 learned_addr learned my addr 172.21.15.37:0/3176932473 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b21171700 10 monclient(hunting): get_auth_request con 0x7f2b1c0a04d0 auth_method 0 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b21171700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b21171700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b21171700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 10 monclient(hunting): get_auth_request con 0x7f2b1c0a2d50 auth_method 0 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b21171700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b21171700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b21171700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 10 monclient(hunting): handle_auth_done global_id 34128 payload 274 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 1 -- 172.21.15.37:0/3176932473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b1c0a04d0 msgr2=0x7f2b1c0a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 1 --2- 172.21.15.37:0/3176932473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b1c0a04d0 0x7f2b1c0a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 1 -- 172.21.15.37:0/3176932473 >> v1:172.21.15.37:6789/0 conn(0x7f2b1c0a0e90 legacy=0x7f2b1c0a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 10 monclient: _finish_hunting 0 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 1 monclient: found mon.noname-a 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 1 -- 172.21.15.37:0/3176932473 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285408368 0 0) 0x7f2b1c0a0900 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 1 -- 172.21.15.37:0/3176932473 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c0a36e0 con 0x7f2b1c0a2d50 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22173700 1 --2- 172.21.15.37:0/3176932473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a2d50 0x7f2b1c0a31a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009d90 tx=0x7f2b1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b764bf95912f008 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 1 -- 172.21.15.37:0/3176932473 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1400e070 con 0x7f2b1c0a2d50 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 10 monclient: dump: 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.964 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 10 monclient: _finish_auth 0 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 10 monclient: _check_auth_tickets 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 1 -- 172.21.15.37:0/3176932473 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1400bb20 con 0x7f2b1c0a2d50 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 5 monclient: authenticate success, global_id 34128 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:53.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 1 -- 172.21.15.37:0/3176932473 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b14012430 con 0x7f2b1c0a2d50 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 4 set_mon_vals no callback set 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b20970700 10 monclient: dump: 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.966 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 10 monclient: shutdown 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 -- 172.21.15.37:0/3176932473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a2d50 msgr2=0x7f2b1c0a31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 --2- 172.21.15.37:0/3176932473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a2d50 0x7f2b1c0a31a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009d90 tx=0x7f2b1400b330 comp rx=0 tx=0).stop 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 -- 172.21.15.37:0/3176932473 shutdown_connections 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 --2- 172.21.15.37:0/3176932473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a2d50 0x7f2b1c0a31a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 --2- 172.21.15.37:0/3176932473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b1c0a04d0 0x7f2b1c0a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 -- 172.21.15.37:0/3176932473 >> 172.21.15.37:0/3176932473 conn(0x7f2b1c0a4740 msgr2=0x7f2b1c0a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 -- 172.21.15.37:0/3176932473 shutdown_connections 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.944+0000 7f2b22974700 1 -- 172.21.15.37:0/3176932473 wait complete. 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient: build_initial_monmap 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient: monmap: 2024-09-16T07:20:53.967 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:53.968 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:53.968 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:53.968 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:53.968 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:53.968 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:53.968 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:53.968 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:53.968 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:53.968 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 1 Processor -- start 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 1 -- start start 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient: init 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b1c0a04d0 0x7f2b1c13bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient(hunting): picked mon.noname-c con 0x7f2b1c0a04d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a0e90 0x7f2b1c13c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient(hunting): picked mon.noname-a con 0x7f2b1c0a0e90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0a2d50 0x7f2b1c122c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient(hunting): picked mon.noname-b con 0x7f2b1c0a2d50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c1268a0 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c1253f0 con 0x7f2b1c0a2d50 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c09dda0 con 0x7f2b1c0a04d0 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient(hunting): authenticate will time out at 974.438424s 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a0e90 0x7f2b1c13c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a0e90 0x7f2b1c13c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55870/0 (socket says 172.21.15.37:55870) 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 1 -- 172.21.15.37:0/3261917854 learned_addr learned my addr 172.21.15.37:0/3261917854 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 10 monclient(hunting): get_auth_request con 0x7f2b1c0a0e90 auth_method 0 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21972700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b1c0a04d0 0x7f2b1c13bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21972700 10 monclient(hunting): get_auth_request con 0x7f2b1c0a04d0 auth_method 0 2024-09-16T07:20:53.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21972700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21972700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21972700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22173700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0a2d50 0x7f2b1c122c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21972700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21972700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21972700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22173700 10 monclient(hunting): get_auth_request con 0x7f2b1c0a2d50 auth_method 0 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22173700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22173700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22173700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 10 monclient(hunting): handle_auth_done global_id 34134 payload 824 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 1 -- 172.21.15.37:0/3261917854 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b1c0a04d0 msgr2=0x7f2b1c13bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b1c0a04d0 0x7f2b1c13bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 1 -- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0a2d50 msgr2=0x7f2b1c122c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0a2d50 0x7f2b1c122c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 10 monclient: _finish_hunting 0 2024-09-16T07:20:53.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 1 monclient: found mon.noname-a 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 1 -- 172.21.15.37:0/3261917854 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b14009a70 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22173700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0a2d50 0x7f2b1c122c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b21171700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a0e90 0x7f2b1c13c170 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800e9d0 tx=0x7f2b1800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b127fc700 1 -- 172.21.15.37:0/3261917854 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18009720 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b127fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b127fc700 10 monclient: dump: 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.973 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b127fc700 10 monclient: _finish_auth 0 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b127fc700 10 monclient: _check_auth_tickets 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 5 monclient: authenticate success, global_id 34134 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b127fc700 1 -- 172.21.15.37:0/3261917854 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1800c850 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b127fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:53.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b127fc700 1 -- 172.21.15.37:0/3261917854 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1800cbc0 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient: _renew_subs 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b22974700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.948+0000 7f2b13fff700 4 set_mon_vals no callback set 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b22974700 1 -- 172.21.15.37:0/3261917854 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c06bd20 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b127fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b127fc700 10 monclient: dump: 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:53.975 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b22974700 10 monclient: _renew_subs 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b22974700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b22974700 1 -- 172.21.15.37:0/3261917854 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1c1265c0 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b22974700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b22974700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b22974700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b127fc700 1 -- 172.21.15.37:0/3261917854 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f2b1800cd20 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b22974700 1 -- 172.21.15.37:0/3261917854 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1c126420 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b127fc700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2b00035600 0x7f2b00037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:53.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b127fc700 1 -- 172.21.15.37:0/3261917854 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f2b18047c60 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b21972700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2b00035600 0x7f2b00037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:53.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b21972700 10 monclient: get_auth_request con 0x7f2b00035600 auth_method 0 2024-09-16T07:20:53.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.952+0000 7f2b21972700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2b00035600 0x7f2b00037ac0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c000f60 tx=0x7f2b0c007e00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:53.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.960+0000 7f2b127fc700 1 -- 172.21.15.37:0/3261917854 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2b1c126420 con 0x7f2b1c0a0e90 2024-09-16T07:20:53.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.960+0000 7f2b127fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:53.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:53.960+0000 7f2b127fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:54.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.200+0000 7f2b22974700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T07:20:54.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.200+0000 7f2b22974700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T07:20:54.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.200+0000 7f2b22974700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:54.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.200+0000 7f2b22974700 1 -- 172.21.15.37:0/3261917854 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f2b1c126420 con 0x7f2b1c0a0e90 2024-09-16T07:20:54.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b127fc700 1 -- 172.21.15.37:0/3261917854 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f2b1c126420 con 0x7f2b1c0a0e90 2024-09-16T07:20:54.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b127fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T07:20:54.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b127fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:54.206 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.207 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=729601666,config=3541143443,health=2244247403,kv=3696321355,logm=3045307257}) 2024-09-16T07:20:54.207 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=1,health=12,kv=12,logm=65} crc {auth=306511960,config=555582061,health=2244247403,kv=3696321355,logm=1648752634}) 2024-09-16T07:20:54.207 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.207 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=729601666,config=3541143443,health=2244247403,kv=3696321355,logm=3045307257}) 2024-09-16T07:20:54.207 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=306511960,config=3541143443,health=2244247403,kv=740148867,logm=2968499836}) 2024-09-16T07:20:54.208 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.208 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4188371089}) 2024-09-16T07:20:54.208 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2897210483}) 2024-09-16T07:20:54.208 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.208 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4188371089}) 2024-09-16T07:20:54.208 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.205+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3779670316}) 2024-09-16T07:20:54.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 -- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2b00035600 msgr2=0x7f2b00037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:54.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2b00035600 0x7f2b00037ac0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c000f60 tx=0x7f2b0c007e00 comp rx=0 tx=0).stop 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 10 monclient: shutdown 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 -- 172.21.15.37:0/3261917854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a0e90 msgr2=0x7f2b1c13c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a0e90 0x7f2b1c13c170 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800e9d0 tx=0x7f2b1800ece0 comp rx=0 tx=0).stop 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 -- 172.21.15.37:0/3261917854 shutdown_connections 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b1c0a2d50 0x7f2b1c122c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b1c0a0e90 0x7f2b1c13c170 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2b00035600 0x7f2b00037ac0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 --2- 172.21.15.37:0/3261917854 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b1c0a04d0 0x7f2b1c13bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 -- 172.21.15.37:0/3261917854 >> 172.21.15.37:0/3261917854 conn(0x7f2b1c0a4740 msgr2=0x7f2b1c09e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 -- 172.21.15.37:0/3261917854 shutdown_connections 2024-09-16T07:20:54.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:54.204+0000 7f2b22974700 1 -- 172.21.15.37:0/3261917854 wait complete. 2024-09-16T07:20:54.211 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.209+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.211 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.209+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1849127102}) 2024-09-16T07:20:54.211 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.209+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3913645332}) 2024-09-16T07:20:54.211 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.209+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.211 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.209+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1849127102}) 2024-09-16T07:20:54.211 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.209+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3761784301}) 2024-09-16T07:20:54.214 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.214 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=542149555}) 2024-09-16T07:20:54.214 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354360098}) 2024-09-16T07:20:54.214 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.214 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=542149555}) 2024-09-16T07:20:54.214 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4036546198}) 2024-09-16T07:20:54.216 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.216 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1346125111}) 2024-09-16T07:20:54.216 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1545740655}) 2024-09-16T07:20:54.217 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.217 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1346125111}) 2024-09-16T07:20:54.217 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.213+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2755796308}) 2024-09-16T07:20:54.218 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.217+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.218 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.217+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=1775664704,mdsmap=1531151339,mgr=261139402}) 2024-09-16T07:20:54.218 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.217+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=2099774022,mdsmap=1531151339,mgr=3869222304}) 2024-09-16T07:20:54.218 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.217+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.218 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.217+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=1775664704,mdsmap=1531151339,mgr=261139402}) 2024-09-16T07:20:54.218 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.217+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=1651072556,mdsmap=1531151339,mgr=2259685124}) 2024-09-16T07:20:54.220 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.221 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=59} crc {mgr=2116737488,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=3741914581,mon_config_key=2895360784,monmap=38151330,osd_metadata=598972236,osd_pg_creating=3597283685,osd_snap=1614000649,osdmap=3065949804}) 2024-09-16T07:20:54.221 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=59} crc {mgr=2116737488,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=3257378281,mon_config_key=2895360784,monmap=1839276543,osd_metadata=598972236,osd_pg_creating=3597283685,osd_snap=1614000649,osdmap=2593021512}) 2024-09-16T07:20:54.221 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.221 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=59} crc {mgr=2116737488,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=3741914581,mon_config_key=2895360784,monmap=38151330,osd_metadata=598972236,osd_pg_creating=3597283685,osd_snap=1614000649,osdmap=3065949804}) 2024-09-16T07:20:54.221 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=12,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=64} crc {mgr=2116737488,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=3249117076,mon_config_key=3895749206,monmap=1839276543,osd_metadata=132974247,osd_pg_creating=3597283685,osd_snap=1614000649,osdmap=1502032525}) 2024-09-16T07:20:54.222 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.222 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3994964423}) 2024-09-16T07:20:54.222 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4134104441}) 2024-09-16T07:20:54.223 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.223 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3994964423}) 2024-09-16T07:20:54.223 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2708145733}) 2024-09-16T07:20:54.223 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.223 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=28} crc {osdmap=990138345}) 2024-09-16T07:20:54.223 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=30} crc {osdmap=3923072008}) 2024-09-16T07:20:54.224 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:20:54.224 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=28} crc {osdmap=990138345}) 2024-09-16T07:20:54.224 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:20:54.221+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=29} crc {osdmap=860488653}) 2024-09-16T07:20:54.225 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T07:20:54.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25685]: starting. 2024-09-16T07:20:54.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25685]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25684 2024-09-16T07:20:54.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25685]: created object 0... 2024-09-16T07:20:54.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25685]: created object 25... 2024-09-16T07:20:54.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25685]: created object 49... 2024-09-16T07:20:54.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25685]: finishing. 2024-09-16T07:20:54.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25685]: shutting down. 2024-09-16T07:20:55.226 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T07:20:55.227 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:20:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient: get_monmap_and_config 2024-09-16T07:20:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient: build_initial_monmap 2024-09-16T07:20:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient: monmap: 2024-09-16T07:20:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:55.348066+0000 2024-09-16T07:20:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:55.348066+0000 2024-09-16T07:20:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:55.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 1 Processor -- start 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 1 -- start start 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient: init 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f023c0a2750 0x7f023c0a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient(hunting): picked mon.noname-c con 0x7f023c0a2750 addr v2:172.21.15.143:3300/0 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f023c0a3070 addr v1:172.21.15.37:6789/0 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f023c133730 addr v1:172.21.15.143:6789/0 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023c099be0 con 0x7f023c0a3070 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023c09de40 con 0x7f023c133730 2024-09-16T07:20:55.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f023c09d530 con 0x7f023c0a2750 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02433a6700 10 monclient(hunting): authenticate will time out at 975.833947s 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f023c0a3070 0x7f023c133020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50936/0 (socket says 172.21.15.37:50936) 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 1 -- 172.21.15.37:0/336468955 learned_addr learned my addr 172.21.15.37:0/336468955 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 1 --2- 172.21.15.37:0/336468955 >> v2:172.21.15.143:3300/0 conn(0x7f023c0a2750 0x7f023c0a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648907302 0 0) 0x7f023c099be0 con 0x7f023c0a3070 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): _negotiate 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): my global_id is 34137 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02280035d0 con 0x7f023c0a3070 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): get_auth_request con 0x7f023c0a2750 auth_method 0 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1168124511 0 0) 0x7f02280035d0 con 0x7f023c0a3070 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 >> v2:172.21.15.143:3300/0 conn(0x7f023c0a2750 msgr2=0x7f023c0a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 --2- 172.21.15.37:0/336468955 >> v2:172.21.15.143:3300/0 conn(0x7f023c0a2750 0x7f023c0a2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 >> v1:172.21.15.143:6789/0 conn(0x7f023c133730 legacy=0x7f023c13b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient: _finish_hunting 0 2024-09-16T07:20:55.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 monclient: found mon.noname-a-legacy 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c09a020 con 0x7f023c0a3070 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient: _finish_auth 0 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient: _check_auth_tickets 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 120013200 0 0) 0x7f0238001670 con 0x7f023c0a3070 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient: dump: 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.362 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 >> v1:172.21.15.37:6789/0 conn(0x7f023c0a3070 legacy=0x7f023c133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0228003a20 0x7f0228003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): picked mon.b con 0x7f0228003a20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0228004870 0x7f0228004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): picked mon.a con 0x7f0228004870 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:55.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0228005350 0x7f022800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): picked mon.c con 0x7f0228005350 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:55.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c09d530 con 0x7f0228004870 2024-09-16T07:20:55.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022800c220 con 0x7f0228003a20 2024-09-16T07:20:55.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02280107f0 con 0x7f0228005350 2024-09-16T07:20:55.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:55.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02413a2700 10 monclient(hunting): _finish_auth 0 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0228004870 0x7f0228004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0242ba5700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0228005350 0x7f022800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 10 monclient(hunting): get_auth_request con 0x7f0228004870 auth_method 0 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0242ba5700 10 monclient(hunting): get_auth_request con 0x7f0228005350 auth_method 0 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0242ba5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0242ba5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0242ba5700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0228003a20 0x7f0228003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): get_auth_request con 0x7f0228003a20 auth_method 0 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0242ba5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0242ba5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0242ba5700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f02423a4700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.344+0000 7f0241ba3700 10 monclient(hunting): handle_auth_done global_id 34137 payload 293 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 1 -- 172.21.15.37:0/336468955 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0228005350 msgr2=0x7f022800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0228005350 0x7f022800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 1 -- 172.21.15.37:0/336468955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0228003a20 msgr2=0x7f0228003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0228003a20 0x7f0228003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 10 monclient: _finish_hunting 0 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 1 monclient: found mon.a 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 1 -- 172.21.15.37:0/336468955 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02280035d0 con 0x7f0228004870 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 10 monclient: _finish_auth 0 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 10 monclient: _check_auth_tickets 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0242ba5700 1 -- 172.21.15.37:0/336468955 reap_dead start 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 5 monclient: authenticate success, global_id 34137 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:55.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f0241ba3700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0228004870 0x7f0228004cf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f023800c2f0 tx=0x7f02280058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6535e335d6629c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238007c30 con 0x7f0228004870 2024-09-16T07:20:55.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02413a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02413a2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:55.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02413a2700 10 monclient: dump: 2024-09-16T07:20:55.368 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023800ce10 con 0x7f0228004870 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02413a2700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:55.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02413a2700 1 -- 172.21.15.37:0/336468955 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238005bc0 con 0x7f0228004870 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02413a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02413a2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02413a2700 10 monclient: dump: 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.370 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 4 set_mon_vals no callback set 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 10 monclient: shutdown 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 1 -- 172.21.15.37:0/336468955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0228004870 msgr2=0x7f0228004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0228004870 0x7f0228004cf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f023800c2f0 tx=0x7f02280058a0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 1 -- 172.21.15.37:0/336468955 shutdown_connections 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 1 --2- 172.21.15.37:0/336468955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0228004870 0x7f0228004cf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 1 -- 172.21.15.37:0/336468955 >> 172.21.15.37:0/336468955 conn(0x7f023c0a3ea0 msgr2=0x7f023c0a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 1 -- 172.21.15.37:0/336468955 shutdown_connections 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 1 -- 172.21.15.37:0/336468955 wait complete. 2024-09-16T07:20:55.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 10 monclient: build_initial_monmap 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.348+0000 7f02433a6700 10 monclient: monmap: 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 1 Processor -- start 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 1 -- start start 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient: init 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0228004870 0x7f023c1303c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient(hunting): picked mon.noname-b con 0x7f0228004870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f023c1c6180 0x7f023c130900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient(hunting): picked mon.noname-c con 0x7f023c1c6180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f023c11be80 0x7f023c11c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient(hunting): picked mon.noname-a con 0x7f023c11be80 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c1c6ea0 con 0x7f023c11be80 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c11f8c0 con 0x7f0228004870 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023c130e40 con 0x7f023c1c6180 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient(hunting): authenticate will time out at 975.841840s 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f023c11be80 0x7f023c11c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0241ba3700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f023c1c6180 0x7f023c130900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f023c11be80 0x7f023c11c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55896/0 (socket says 172.21.15.37:55896) 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 1 -- 172.21.15.37:0/1278499913 learned_addr learned my addr 172.21.15.37:0/1278499913 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02423a4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0228004870 0x7f023c1303c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 10 monclient(hunting): get_auth_request con 0x7f023c11be80 auth_method 0 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0241ba3700 10 monclient(hunting): get_auth_request con 0x7f023c1c6180 auth_method 0 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0241ba3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0241ba3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0241ba3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02423a4700 10 monclient(hunting): get_auth_request con 0x7f0228004870 auth_method 0 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02423a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02423a4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02423a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0241ba3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0241ba3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0241ba3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02423a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02423a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02423a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 10 monclient(hunting): handle_auth_done global_id 34140 payload 824 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 1 -- 172.21.15.37:0/1278499913 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f023c1c6180 msgr2=0x7f023c130900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f023c1c6180 0x7f023c130900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 1 -- 172.21.15.37:0/1278499913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0228004870 msgr2=0x7f023c1303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0228004870 0x7f023c1303c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 10 monclient: _finish_hunting 0 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 1 monclient: found mon.noname-a 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 1 -- 172.21.15.37:0/1278499913 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c09b250 con 0x7f023c11be80 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0242ba5700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f023c11be80 0x7f023c11c2e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f023c069c20 tx=0x7f023400d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 1 -- 172.21.15.37:0/1278499913 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0234003c30 con 0x7f023c11be80 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 10 monclient: dump: 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.377 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 10 monclient: _finish_auth 0 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 5 monclient: authenticate success, global_id 34140 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient: _renew_subs 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 1 -- 172.21.15.37:0/1278499913 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0234003db0 con 0x7f023c11be80 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:55.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 1 -- 172.21.15.37:0/1278499913 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023c130fe0 con 0x7f023c11be80 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 1 -- 172.21.15.37:0/1278499913 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023400d950 con 0x7f023c11be80 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0232ffd700 10 monclient: dump: 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.379 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient: _renew_subs 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f02433a6700 1 -- 172.21.15.37:0/1278499913 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023c0724c0 con 0x7f023c11be80 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.352+0000 7f0240ba1700 4 set_mon_vals no callback set 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.356+0000 7f0232ffd700 1 -- 172.21.15.37:0/1278499913 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f023400dab0 con 0x7f023c11be80 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.356+0000 7f0232ffd700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f0220035600 0x7f0220037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.356+0000 7f0232ffd700 1 -- 172.21.15.37:0/1278499913 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f0234047cd0 con 0x7f023c11be80 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.356+0000 7f02433a6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.356+0000 7f02433a6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.356+0000 7f02433a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:55.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.356+0000 7f02433a6700 1 -- 172.21.15.37:0/1278499913 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023c066a10 con 0x7f023c11be80 2024-09-16T07:20:55.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.356+0000 7f02423a4700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f0220035600 0x7f0220037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.356+0000 7f02423a4700 10 monclient: get_auth_request con 0x7f0220035600 auth_method 0 2024-09-16T07:20:55.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.356+0000 7f02423a4700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f0220035600 0x7f0220037ac0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f023c02ce60 tx=0x7f022c009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.360+0000 7f0232ffd700 1 -- 172.21.15.37:0/1278499913 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f023401d460 con 0x7f023c11be80 2024-09-16T07:20:55.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.360+0000 7f0232ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:55.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.360+0000 7f0232ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:55.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.592+0000 7f02433a6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:20:55.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.592+0000 7f02433a6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:20:55.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.592+0000 7f02433a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:55.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.592+0000 7f02433a6700 1 -- 172.21.15.37:0/1278499913 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f023c133ba0 con 0x7f023c11be80 2024-09-16T07:20:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.592+0000 7f0232ffd700 1 -- 172.21.15.37:0/1278499913 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f023403b030 con 0x7f023c11be80 2024-09-16T07:20:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.596+0000 7f0232ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:20:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.596+0000 7f0232ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:55.598 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.596+0000 7f02433a6700 1 -- 172.21.15.37:0/1278499913 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f0220035600 msgr2=0x7f0220037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.596+0000 7f02433a6700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f0220035600 0x7f0220037ac0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f023c02ce60 tx=0x7f022c009240 comp rx=0 tx=0).stop 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.596+0000 7f02433a6700 10 monclient: shutdown 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.596+0000 7f02433a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.596+0000 7f02433a6700 1 -- 172.21.15.37:0/1278499913 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f023c11be80 msgr2=0x7f023c11c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.596+0000 7f02433a6700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f023c11be80 0x7f023c11c2e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f023c069c20 tx=0x7f023400d150 comp rx=0 tx=0).stop 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.596+0000 7f02433a6700 1 -- 172.21.15.37:0/1278499913 shutdown_connections 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.596+0000 7f02433a6700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f023c11be80 0x7f023c11c2e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.600+0000 7f02433a6700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f0220035600 0x7f0220037ac0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.600+0000 7f02433a6700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f023c1c6180 0x7f023c130900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.600+0000 7f02433a6700 1 --2- 172.21.15.37:0/1278499913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0228004870 0x7f023c1303c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.600+0000 7f02433a6700 1 -- 172.21.15.37:0/1278499913 >> 172.21.15.37:0/1278499913 conn(0x7f023c0a3ea0 msgr2=0x7f023c09f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.600+0000 7f02433a6700 1 -- 172.21.15.37:0/1278499913 shutdown_connections 2024-09-16T07:20:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.600+0000 7f02433a6700 1 -- 172.21.15.37:0/1278499913 wait complete. 2024-09-16T07:20:55.617 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T07:20:55.618 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T07:20:55.618 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T07:20:55.618 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T07:20:55.703 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25686]: starting. 2024-09-16T07:20:55.703 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25686]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25684 2024-09-16T07:20:55.703 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25686]: shutting down. 2024-09-16T07:20:55.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient: get_monmap_and_config 2024-09-16T07:20:55.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient: build_initial_monmap 2024-09-16T07:20:55.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient: monmap: 2024-09-16T07:20:55.731 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:55.731 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.731 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:55.732451+0000 2024-09-16T07:20:55.731 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:55.732451+0000 2024-09-16T07:20:55.731 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:55.732 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.732 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:55.732 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:55.732 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:55.732 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:55.732 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:55.732 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 1 Processor -- start 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 1 -- start start 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient: init 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024020 0x7f9510024400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient(hunting): picked mon.noname-b con 0x7f9510024020 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f95100bb920 0x7f95100bbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient(hunting): picked mon.noname-a con 0x7f95100bb920 addr v2:172.21.15.37:3300/0 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9510024f80 0x7f95100253b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient(hunting): picked mon.noname-c con 0x7f9510024f80 addr v2:172.21.15.143:3300/0 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9510027960 con 0x7f95100bb920 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9510027f20 con 0x7f9510024f80 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9510027680 con 0x7f9510024020 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient(hunting): authenticate will time out at 976.218665s 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95177fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024020 0x7f9510024400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f95100bb920 0x7f95100bbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f95100bb920 0x7f95100bbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55904/0 (socket says 172.21.15.37:55904) 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95177fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024020 0x7f9510024400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38150/0 (socket says 172.21.15.37:38150) 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 1 -- 172.21.15.37:0/2819237408 learned_addr learned my addr 172.21.15.37:0/2819237408 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:55.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 10 monclient(hunting): get_auth_request con 0x7f95100bb920 auth_method 0 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95177fe700 10 monclient(hunting): get_auth_request con 0x7f9510024020 auth_method 0 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 10 monclient(hunting): handle_auth_done global_id 34167 payload 274 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 1 -- 172.21.15.37:0/2819237408 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024020 msgr2=0x7f9510024400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 1 --2- 172.21.15.37:0/2819237408 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024020 0x7f9510024400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 1 -- 172.21.15.37:0/2819237408 >> v2:172.21.15.143:3300/0 conn(0x7f9510024f80 msgr2=0x7f95100253b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9517fff700 1 --2- 172.21.15.37:0/2819237408 >> v2:172.21.15.143:3300/0 conn(0x7f9510024f80 0x7f95100253b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 1 --2- 172.21.15.37:0/2819237408 >> v2:172.21.15.143:3300/0 conn(0x7f9510024f80 0x7f95100253b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 10 monclient: _finish_hunting 0 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 1 monclient: found mon.noname-a 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 1 -- 172.21.15.37:0/2819237408 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9510027c40 con 0x7f95100bb920 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95177fe700 1 --2- 172.21.15.37:0/2819237408 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024020 0x7f9510024400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f9516ffd700 1 --2- 172.21.15.37:0/2819237408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100bb920 0x7f95100bbd00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f951002a3f0 tx=0x7f9504009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c8007e2c10ea1b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 1 -- 172.21.15.37:0/2819237408 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504011830 con 0x7f95100bb920 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 10 monclient: dump: 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.740 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 10 monclient: _finish_auth 0 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 10 monclient: _check_auth_tickets 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 5 monclient: authenticate success, global_id 34167 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 1 -- 172.21.15.37:0/2819237408 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9504011990 con 0x7f95100bb920 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f951cb25700 4 set_mon_vals no callback set 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 1 -- 172.21.15.37:0/2819237408 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504011d20 con 0x7f95100bb920 2024-09-16T07:20:55.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.728+0000 7f95167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95167fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95167fc700 10 monclient: dump: 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient: shutdown 2024-09-16T07:20:55.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 -- 172.21.15.37:0/2819237408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100bb920 msgr2=0x7f95100bbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 --2- 172.21.15.37:0/2819237408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100bb920 0x7f95100bbd00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f951002a3f0 tx=0x7f9504009bf0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 -- 172.21.15.37:0/2819237408 shutdown_connections 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 --2- 172.21.15.37:0/2819237408 >> v2:172.21.15.143:3300/0 conn(0x7f9510024f80 0x7f95100253b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 --2- 172.21.15.37:0/2819237408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100bb920 0x7f95100bbd00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 --2- 172.21.15.37:0/2819237408 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024020 0x7f9510024400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 -- 172.21.15.37:0/2819237408 >> 172.21.15.37:0/2819237408 conn(0x7f9510006ce0 msgr2=0x7f9510028520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 -- 172.21.15.37:0/2819237408 shutdown_connections 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 -- 172.21.15.37:0/2819237408 wait complete. 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient: build_initial_monmap 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient: monmap: 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:55.743 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:55.744 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:55.744 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:55.744 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:55.744 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:55.744 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:55.744 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:55.744 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 Processor -- start 2024-09-16T07:20:55.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 -- start start 2024-09-16T07:20:55.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient: init 2024-09-16T07:20:55.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:55.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:20:55.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9510024020 0x7f9510150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient(hunting): picked mon.noname-a con 0x7f9510024020 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:55.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024f80 0x7f9510150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient(hunting): picked mon.noname-c con 0x7f9510024f80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:55.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95100bb920 0x7f95100a5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient(hunting): picked mon.noname-b con 0x7f95100bb920 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:55.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95100a85b0 con 0x7f9510024020 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95100bb100 con 0x7f95100bb920 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95100234b0 con 0x7f9510024f80 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f951cb25700 10 monclient(hunting): authenticate will time out at 976.222871s 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9510024020 0x7f9510150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9510024020 0x7f9510150800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55908/0 (socket says 172.21.15.37:55908) 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 1 -- 172.21.15.37:0/1512841679 learned_addr learned my addr 172.21.15.37:0/1512841679 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 10 monclient(hunting): get_auth_request con 0x7f9510024020 auth_method 0 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9516ffd700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024f80 0x7f9510150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9516ffd700 10 monclient(hunting): get_auth_request con 0x7f9510024f80 auth_method 0 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9516ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9516ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9516ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9517fff700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95100bb920 0x7f95100a5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9517fff700 10 monclient(hunting): get_auth_request con 0x7f95100bb920 auth_method 0 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9517fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9517fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9517fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 10 monclient(hunting): handle_auth_done global_id 34170 payload 824 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 1 -- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024f80 msgr2=0x7f9510150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9510024f80 0x7f9510150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 1 -- 172.21.15.37:0/1512841679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95100bb920 msgr2=0x7f95100a5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95100bb920 0x7f95100a5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 10 monclient: _finish_hunting 0 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 1 monclient: found mon.noname-a 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 1 -- 172.21.15.37:0/1512841679 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c0094b0 con 0x7f9510024020 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9517fff700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95100bb920 0x7f95100a5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f95177fe700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9510024020 0x7f9510150800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f950c00ebe0 tx=0x7f950c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c00c890 con 0x7f9510024020 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9503fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9503fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9503fff700 10 monclient: dump: 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.753 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9503fff700 10 monclient: _finish_auth 0 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9503fff700 10 monclient: _check_auth_tickets 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.732+0000 7f9503fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 5 monclient: authenticate success, global_id 34170 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950c00c9f0 con 0x7f9510024020 2024-09-16T07:20:55.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 10 monclient: _renew_subs 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 1 -- 172.21.15.37:0/1512841679 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9504009570 con 0x7f9510024020 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c00cd80 con 0x7f9510024020 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9515ffb700 4 set_mon_vals no callback set 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 10 monclient: dump: 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.755 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 10 monclient: _renew_subs 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 1 -- 172.21.15.37:0/1512841679 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95100a9a60 con 0x7f9510024020 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6310 0x7f95100a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f951cb25700 1 -- 172.21.15.37:0/1512841679 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9510007f90 con 0x7f95100a6310 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6310 0x7f95100a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 10 monclient: get_auth_request con 0x7f95100a6310 auth_method 0 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f950c013470 con 0x7f9510024020 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 10 monclient(hunting): handle_auth_done global_id 34173 payload 824 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 1 -- 172.21.15.37:0/1512841679 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95100bc240 con 0x7f95100a6310 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9516ffd700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6310 0x7f95100a66f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f951002a3f0 tx=0x7f9504004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f94fc035600 0x7f94fc037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f950c047bc0 con 0x7f9510024020 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504020070 con 0x7f95100a6310 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9517fff700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f94fc035600 0x7f94fc037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 10 monclient: dump: 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9517fff700 10 monclient: get_auth_request con 0x7f94fc035600 auth_method 0 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9517fff700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f94fc035600 0x7f94fc037ac0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f951806a1b0 tx=0x7f951806e040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f9504004310 con 0x7f95100a6310 2024-09-16T07:20:55.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:55.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:55.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6310 msgr2=0x7f95100a66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6310 0x7f95100a66f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f951002a3f0 tx=0x7f9504004770 comp rx=0 tx=0).stop 2024-09-16T07:20:55.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6310 msgr2=0x7f95100a66f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:55.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.736+0000 7f9503fff700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6310 0x7f95100a66f0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f951002a3f0 tx=0x7f9504004770 comp rx=0 tx=0).stop 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6c30 0x7f9510002970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 -- 172.21.15.37:0/1512841679 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95100253b0 con 0x7f95100a6c30 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6c30 0x7f9510002970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 10 monclient: get_auth_request con 0x7f95100a6c30 auth_method 0 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 10 monclient(hunting): handle_auth_done global_id 34176 payload 824 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 1 -- 172.21.15.37:0/1512841679 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f95100008d0 con 0x7f95100a6c30 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9516ffd700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6c30 0x7f9510002970 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9504009150 tx=0x7f9504011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950402d8d0 con 0x7f95100a6c30 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 10 monclient: dump: 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.762 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4453 (secure 0 0 0) 0x7f9504031070 con 0x7f95100a6c30 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6c30 msgr2=0x7f9510002970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6c30 0x7f9510002970 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9504009150 tx=0x7f9504011000 comp rx=0 tx=0).stop 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 1 -- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6c30 msgr2=0x7f9510002970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:55.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9503fff700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95100a6c30 0x7f9510002970 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9504009150 tx=0x7f9504011000 comp rx=0 tx=0).stop 2024-09-16T07:20:55.764 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:20:55.764 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:20:55.764 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:20:55.764 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:20:55.764 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 16, 2024-09-16T07:20:55.764 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:20:55.764 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:20:55.764 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:55.765 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:55.766 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:55.767 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:55.768 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:55.769 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:55.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:55.769 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:55.769 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:55.769 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:55.769 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:55.769 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:55.770 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:55.770 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:55.770 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:20:55.770 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.770 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:20:55.770 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.770 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:20:55.770 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.771 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.772 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.772 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.772 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:20:55.772 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:20:55.772 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:55.772 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:55.772 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:55.772 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.772 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.772 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:20:55.773 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:20:55.774 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.775 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.776 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.777 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:20:55.778 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:20:55.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 -- 172.21.15.37:0/1512841679 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f94fc035600 msgr2=0x7f94fc037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f94fc035600 0x7f94fc037ac0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f951806a1b0 tx=0x7f951806e040 comp rx=0 tx=0).stop 2024-09-16T07:20:55.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 10 monclient: shutdown 2024-09-16T07:20:55.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:55.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 -- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9510024020 msgr2=0x7f9510150800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f9517fff700 1 -- 172.21.15.37:0/1512841679 reap_dead start 2024-09-16T07:20:55.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9510024020 0x7f9510150800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f950c00ebe0 tx=0x7f950c00eef0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 -- 172.21.15.37:0/1512841679 shutdown_connections 2024-09-16T07:20:55.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 --2- 172.21.15.37:0/1512841679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9510024020 0x7f9510150800 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 -- 172.21.15.37:0/1512841679 >> 172.21.15.37:0/1512841679 conn(0x7f9510006ce0 msgr2=0x7f9510021cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:55.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 -- 172.21.15.37:0/1512841679 shutdown_connections 2024-09-16T07:20:55.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.748+0000 7f951cb25700 1 -- 172.21.15.37:0/1512841679 wait complete. 2024-09-16T07:20:55.780 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:20:55.780 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T07:20:55.780 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T07:20:55.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient: get_monmap_and_config 2024-09-16T07:20:55.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient: build_initial_monmap 2024-09-16T07:20:55.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient: monmap: 2024-09-16T07:20:55.904 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:55.904 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.904 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:55.905048+0000 2024-09-16T07:20:55.904 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:55.905048+0000 2024-09-16T07:20:55.904 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:55.904 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.904 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:55.905 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:55.905 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:55.905 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:55.905 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:55.905 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 1 Processor -- start 2024-09-16T07:20:55.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 1 -- start start 2024-09-16T07:20:55.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient: init 2024-09-16T07:20:55.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:55.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T07:20:55.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c09f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient(hunting): picked mon.noname-b con 0x7f6d5c09f3c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:55.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d5c09c230 addr v1:172.21.15.37:6789/0 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d5c09fce0 addr v1:172.21.15.143:6789/0 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5c09b570 con 0x7f6d5c09c230 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5c09baa0 con 0x7f6d5c09fce0 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c09bd80 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d61177700 10 monclient(hunting): authenticate will time out at 976.391028s 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c09f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5259c700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6d5c09c230 0x7f6d5c139980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50950/0 (socket says 172.21.15.37:50950) 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5259c700 1 -- 172.21.15.37:0/4119674902 learned_addr learned my addr 172.21.15.37:0/4119674902 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:20:55.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 10 monclient(hunting): get_auth_request con 0x7f6d5c09f3c0 auth_method 0 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 10 monclient(hunting): handle_auth_done global_id 4841 payload 274 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 1 -- 172.21.15.37:0/4119674902 >> v1:172.21.15.143:6789/0 conn(0x7f6d5c09fce0 legacy=0x7f6d5c13a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 1 -- 172.21.15.37:0/4119674902 >> v1:172.21.15.37:6789/0 conn(0x7f6d5c09c230 legacy=0x7f6d5c139980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 10 monclient: _finish_hunting 0 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 1 monclient: found mon.noname-b 2024-09-16T07:20:55.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5ad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.900+0000 7f6d5a59c700 1 -- 172.21.15.37:0/4119674902 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303797095 0 0) 0x7f6d5c09b570 con 0x7f6d5c09c230 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5ad9d700 1 -- 172.21.15.37:0/4119674902 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c09a230 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5ad9d700 1 --2- 172.21.15.37:0/4119674902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c09f7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009880 tx=0x7f6d4c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f6306d61f46864 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 1 -- 172.21.15.37:0/4119674902 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c011830 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 10 monclient: dump: 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.918 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 10 monclient: _finish_auth 0 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 10 monclient: _check_auth_tickets 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 5 monclient: authenticate success, global_id 4841 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:55.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 1 -- 172.21.15.37:0/4119674902 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c011990 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 4 set_mon_vals no callback set 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 1 -- 172.21.15.37:0/4119674902 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c011d20 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5a59c700 10 monclient: dump: 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.920 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.921 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.921 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.921 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.921 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.921 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.921 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient: shutdown 2024-09-16T07:20:55.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:55.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 -- 172.21.15.37:0/4119674902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 msgr2=0x7f6d5c09f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 --2- 172.21.15.37:0/4119674902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c09f7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009880 tx=0x7f6d4c009b90 comp rx=0 tx=0).stop 2024-09-16T07:20:55.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 -- 172.21.15.37:0/4119674902 shutdown_connections 2024-09-16T07:20:55.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 --2- 172.21.15.37:0/4119674902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c09f7a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 -- 172.21.15.37:0/4119674902 >> 172.21.15.37:0/4119674902 conn(0x7f6d5c0a04b0 msgr2=0x7f6d5c0a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:55.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 -- 172.21.15.37:0/4119674902 shutdown_connections 2024-09-16T07:20:55.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 -- 172.21.15.37:0/4119674902 wait complete. 2024-09-16T07:20:55.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient: build_initial_monmap 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient: monmap: 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:55.923 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 Processor -- start 2024-09-16T07:20:55.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 -- start start 2024-09-16T07:20:55.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient: init 2024-09-16T07:20:55.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:55.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:20:55.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d5c09c230 0x7f6d5c1ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient(hunting): picked mon.noname-b con 0x7f6d5c09c230 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:55.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c1ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient(hunting): picked mon.noname-c con 0x7f6d5c09f3c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:55.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d5c09fce0 0x7f6d5c126110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient(hunting): picked mon.noname-a con 0x7f6d5c09fce0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:55.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c139060 con 0x7f6d5c09fce0 2024-09-16T07:20:55.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c072690 con 0x7f6d5c09c230 2024-09-16T07:20:55.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c128250 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:55.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d61177700 10 monclient(hunting): authenticate will time out at 976.395220s 2024-09-16T07:20:55.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c1ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5b59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d5c09fce0 0x7f6d5c126110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c1ceed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38170/0 (socket says 172.21.15.37:38170) 2024-09-16T07:20:55.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 1 -- 172.21.15.37:0/1378857694 learned_addr learned my addr 172.21.15.37:0/1378857694 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:55.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 10 monclient(hunting): get_auth_request con 0x7f6d5c09f3c0 auth_method 0 2024-09-16T07:20:55.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5b59e700 10 monclient(hunting): get_auth_request con 0x7f6d5c09fce0 auth_method 0 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5b59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5b59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5ad9d700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d5c09c230 0x7f6d5c1ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5ad9d700 10 monclient(hunting): get_auth_request con 0x7f6d5c09c230 auth_method 0 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 10 monclient(hunting): handle_auth_done global_id 4844 payload 824 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 1 -- 172.21.15.37:0/1378857694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d5c09c230 msgr2=0x7f6d5c1ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d5c09c230 0x7f6d5c1ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 1 -- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d5c09fce0 msgr2=0x7f6d5c126110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d5c09fce0 0x7f6d5c126110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 10 monclient: _finish_hunting 0 2024-09-16T07:20:55.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 1 monclient: found mon.noname-c 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5259c700 1 -- 172.21.15.37:0/1378857694 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c0094b0 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.904+0000 7f6d5b59e700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d5c09fce0 0x7f6d5c126110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5259c700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c1ceed0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400c550 tx=0x7f6d4400c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d44004ca0 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: dump: 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.929 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.930 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.930 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.930 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.930 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.930 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.930 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.930 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.930 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: _finish_auth 0 2024-09-16T07:20:55.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: _check_auth_tickets 2024-09-16T07:20:55.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:55.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 5 monclient: authenticate success, global_id 4844 2024-09-16T07:20:55.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 10 monclient: _renew_subs 2024-09-16T07:20:55.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d44004e00 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:55.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 1 -- 172.21.15.37:0/1378857694 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5c129d50 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:55.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d44007310 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: dump: 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.932 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d59d9b700 4 set_mon_vals no callback set 2024-09-16T07:20:55.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 10 monclient: _renew_subs 2024-09-16T07:20:55.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:20:55.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 1 -- 172.21.15.37:0/1378857694 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5c1288a0 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f6d44007470 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:55.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6d3c035630 0x7f6d3c037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126650 0x7f6d5c126a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f6d44047740 con 0x7f6d5c09f3c0 2024-09-16T07:20:55.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d61177700 1 -- 172.21.15.37:0/1378857694 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c139ad0 con 0x7f6d5c126650 2024-09-16T07:20:55.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126650 0x7f6d5c126a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 10 monclient: get_auth_request con 0x7f6d5c126650 auth_method 0 2024-09-16T07:20:55.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5ad9d700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6d3c035630 0x7f6d3c037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5ad9d700 10 monclient: get_auth_request con 0x7f6d3c035630 auth_method 0 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_done global_id 4847 payload 824 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 1 -- 172.21.15.37:0/1378857694 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d5c12a1a0 con 0x7f6d5c126650 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5b59e700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126650 0x7f6d5c126a30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400d3a0 tx=0x7f6d5400d760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54003a20 con 0x7f6d5c126650 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: dump: 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.935 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.936 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.936 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.936 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.936 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.936 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.936 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d5ad9d700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6d3c035630 0x7f6d3c037af0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0090b0 tx=0x7f6d4c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f6d54003b80 con 0x7f6d5c126650 2024-09-16T07:20:55.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:55.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:55.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126650 msgr2=0x7f6d5c126a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126650 0x7f6d5c126a30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400d3a0 tx=0x7f6d5400d760 comp rx=0 tx=0).stop 2024-09-16T07:20:55.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126650 msgr2=0x7f6d5c126a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:55.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.908+0000 7f6d53fff700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126650 0x7f6d5c126a30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d61177700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:20:55.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d61177700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126f70 0x7f6d5c122810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:55.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d61177700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:55.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d61177700 1 -- 172.21.15.37:0/1378857694 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c0675e0 con 0x7f6d5c126f70 2024-09-16T07:20:55.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126f70 0x7f6d5c122810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:55.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 10 monclient: get_auth_request con 0x7f6d5c126f70 auth_method 0 2024-09-16T07:20:55.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:55.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:55.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:55.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 10 monclient(hunting): handle_auth_done global_id 4850 payload 824 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 1 -- 172.21.15.37:0/1378857694 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6d5c071d80 con 0x7f6d5c126f70 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d5b59e700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126f70 0x7f6d5c122810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d54016000 tx=0x7f6d540014a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54023070 con 0x7f6d5c126f70 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.920+0000 7f6d53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d53fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d53fff700 10 monclient: dump: 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:55.940 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4619 (secure 0 0 0) 0x7f6d5401e430 con 0x7f6d5c126f70 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d53fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d53fff700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:55.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126f70 msgr2=0x7f6d5c122810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d53fff700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126f70 0x7f6d5c122810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d54016000 tx=0x7f6d540014a0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d53fff700 1 -- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126f70 msgr2=0x7f6d5c122810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:55.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d53fff700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c126f70 0x7f6d5c122810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.942 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:20:55.942 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:20:55.942 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:20:55.942 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 16, 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:20:55.943 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:55.944 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:55.945 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:20:55.946 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:20:55.947 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:20:55.947 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:20:55.947 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:20:55.947 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:20:55.947 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:20:55.947 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:55.947 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:20:55.947 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:55.947 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:55.948 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:55.948 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:55.948 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:55.948 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:55.948 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:55.948 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:55.948 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:55.948 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:55.948 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:55.949 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:20:55.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.949 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:20:55.949 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.949 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:20:55.949 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:20:55.949 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:55.949 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:55.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.950 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:55.950 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:20:55.950 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.950 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:55.950 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:20:55.950 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.950 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:20:55.951 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.952 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:55.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.955 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:55.956 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:55.956 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:20:55.956 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.956 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:55.956 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:20:55.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.956 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:55.956 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:55.956 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-09-16T07:20:55.956 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.957 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:55.957 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:20:55.957 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:55.957 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:55.957 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:55.957 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:20:55.957 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:55.957 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:55.957 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:55.957 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:20:55.958 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 1 -- 172.21.15.37:0/1378857694 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6d3c035630 msgr2=0x7f6d3c037af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6d3c035630 0x7f6d3c037af0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0090b0 tx=0x7f6d4c008030 comp rx=0 tx=0).stop 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 10 monclient: shutdown 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d5b59e700 1 -- 172.21.15.37:0/1378857694 reap_dead start 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 1 -- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 msgr2=0x7f6d5c1ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c1ceed0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400c550 tx=0x7f6d4400c860 comp rx=0 tx=0).stop 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 1 -- 172.21.15.37:0/1378857694 shutdown_connections 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 1 --2- 172.21.15.37:0/1378857694 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d5c09f3c0 0x7f6d5c1ceed0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 1 -- 172.21.15.37:0/1378857694 >> 172.21.15.37:0/1378857694 conn(0x7f6d5c0a04b0 msgr2=0x7f6d5c139f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:55.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 1 -- 172.21.15.37:0/1378857694 shutdown_connections 2024-09-16T07:20:55.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:55.924+0000 7f6d61177700 1 -- 172.21.15.37:0/1378857694 wait complete. 2024-09-16T07:20:55.961 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:20:55.961 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T07:20:55.961 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T07:20:56.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.080+0000 7f2e36543700 10 monclient: get_monmap_and_config 2024-09-16T07:20:56.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.080+0000 7f2e36543700 10 monclient: build_initial_monmap 2024-09-16T07:20:56.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.080+0000 7f2e36543700 10 monclient: monmap: 2024-09-16T07:20:56.085 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:56.086848+0000 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:56.086848+0000 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:56.086 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 Processor -- start 2024-09-16T07:20:56.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 -- start start 2024-09-16T07:20:56.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient: init 2024-09-16T07:20:56.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:56.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T07:20:56.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e300a2750 addr v1:172.21.15.143:6789/0 2024-09-16T07:20:56.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2e300a3240 0x7f2e301382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient(hunting): picked mon.noname-a con 0x7f2e300a3240 addr v2:172.21.15.37:3300/0 2024-09-16T07:20:56.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e30138830 0x7f2e3013ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient(hunting): picked mon.noname-b con 0x7f2e30138830 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3009d810 con 0x7f2e300a2750 2024-09-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e30099be0 con 0x7f2e300a3240 2024-09-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3009de40 con 0x7f2e30138830 2024-09-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient(hunting): authenticate will time out at 976.572746s 2024-09-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2e300a3240 0x7f2e301382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2e300a3240 0x7f2e301382f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55944/0 (socket says 172.21.15.37:55944) 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 1 -- 172.21.15.37:0/243214132 learned_addr learned my addr 172.21.15.37:0/243214132 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 10 monclient(hunting): get_auth_request con 0x7f2e300a3240 auth_method 0 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e35d42700 1 --2- 172.21.15.37:0/243214132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e30138830 0x7f2e3013ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e35d42700 10 monclient(hunting): get_auth_request con 0x7f2e30138830 auth_method 0 2024-09-16T07:20:56.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e35d42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e35d42700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e35d42700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 10 monclient(hunting): handle_auth_done global_id 34185 payload 274 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 1 -- 172.21.15.37:0/243214132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e30138830 msgr2=0x7f2e3013ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 1 --2- 172.21.15.37:0/243214132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e30138830 0x7f2e3013ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 1 -- 172.21.15.37:0/243214132 >> v1:172.21.15.143:6789/0 conn(0x7f2e300a2750 legacy=0x7f2e300a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 10 monclient: _finish_hunting 0 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 1 monclient: found mon.noname-a 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 1 -- 172.21.15.37:0/243214132 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3009d530 con 0x7f2e300a3240 2024-09-16T07:20:56.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e35d42700 1 --2- 172.21.15.37:0/243214132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e30138830 0x7f2e3013ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e34d40700 1 --2- 172.21.15.37:0/243214132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 0x7f2e301382f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2e20009910 tx=0x7f2e20009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14ae78c74274bb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 1 -- 172.21.15.37:0/243214132 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2000ba60 con 0x7f2e300a3240 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 10 monclient: dump: 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:56.095 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 10 monclient: _finish_auth 0 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 10 monclient: _check_auth_tickets 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 1 -- 172.21.15.37:0/243214132 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2000bbc0 con 0x7f2e300a3240 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 1 -- 172.21.15.37:0/243214132 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20009e60 con 0x7f2e300a3240 2024-09-16T07:20:56.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 5 monclient: authenticate success, global_id 34185 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 4 set_mon_vals no callback set 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e27fff700 10 monclient: dump: 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 10 monclient: shutdown 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 -- 172.21.15.37:0/243214132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 msgr2=0x7f2e301382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 --2- 172.21.15.37:0/243214132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 0x7f2e301382f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2e20009910 tx=0x7f2e20009c20 comp rx=0 tx=0).stop 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 -- 172.21.15.37:0/243214132 shutdown_connections 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 --2- 172.21.15.37:0/243214132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e30138830 0x7f2e3013ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 --2- 172.21.15.37:0/243214132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 0x7f2e301382f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 -- 172.21.15.37:0/243214132 >> 172.21.15.37:0/243214132 conn(0x7f2e300a3ea0 msgr2=0x7f2e300a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 -- 172.21.15.37:0/243214132 shutdown_connections 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.084+0000 7f2e36543700 1 -- 172.21.15.37:0/243214132 wait complete. 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient: build_initial_monmap 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient: monmap: 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:56.099 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:56.100 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:56.100 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 1 Processor -- start 2024-09-16T07:20:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 1 -- start start 2024-09-16T07:20:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient: init 2024-09-16T07:20:56.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:56.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:20:56.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e300a2750 0x7f2e301c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient(hunting): picked mon.noname-c con 0x7f2e300a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:56.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 0x7f2e30137e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient(hunting): picked mon.noname-a con 0x7f2e300a3240 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:56.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e30138830 0x7f2e301c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient(hunting): picked mon.noname-b con 0x7f2e30138830 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:56.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30123f90 con 0x7f2e300a3240 2024-09-16T07:20:56.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3006fea0 con 0x7f2e30138830 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3006d310 con 0x7f2e300a2750 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e36543700 10 monclient(hunting): authenticate will time out at 976.578884s 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 0x7f2e30137e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35541700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e300a2750 0x7f2e301c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 0x7f2e30137e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55960/0 (socket says 172.21.15.37:55960) 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 1 -- 172.21.15.37:0/3853055398 learned_addr learned my addr 172.21.15.37:0/3853055398 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 10 monclient(hunting): get_auth_request con 0x7f2e300a3240 auth_method 0 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:56.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35541700 10 monclient(hunting): get_auth_request con 0x7f2e300a2750 auth_method 0 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35541700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35541700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35541700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35d42700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e30138830 0x7f2e301c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35d42700 10 monclient(hunting): get_auth_request con 0x7f2e30138830 auth_method 0 2024-09-16T07:20:56.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35d42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:56.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35d42700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:56.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35d42700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:56.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35541700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:56.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35541700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:56.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35541700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:56.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 10 monclient(hunting): handle_auth_done global_id 34188 payload 824 2024-09-16T07:20:56.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 1 -- 172.21.15.37:0/3853055398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e300a2750 msgr2=0x7f2e301c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e300a2750 0x7f2e301c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 1 -- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e30138830 msgr2=0x7f2e301c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e30138830 0x7f2e301c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 10 monclient: _finish_hunting 0 2024-09-16T07:20:56.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 1 monclient: found mon.noname-a 2024-09-16T07:20:56.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:56.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:56.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 1 -- 172.21.15.37:0/3853055398 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e20009570 con 0x7f2e300a3240 2024-09-16T07:20:56.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e35d42700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e30138830 0x7f2e301c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:20:56.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.088+0000 7f2e34d40700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 0x7f2e30137e60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e3006b870 tx=0x7f2e20004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:56.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20004300 con 0x7f2e300a3240 2024-09-16T07:20:56.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: dump: 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:56.111 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:56.112 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: _finish_auth 0 2024-09-16T07:20:56.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:20:56.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:56.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 5 monclient: authenticate success, global_id 34188 2024-09-16T07:20:56.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 10 monclient: _renew_subs 2024-09-16T07:20:56.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:56.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 1 -- 172.21.15.37:0/3853055398 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e300724c0 con 0x7f2e300a3240 2024-09-16T07:20:56.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e20023070 con 0x7f2e300a3240 2024-09-16T07:20:56.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:56.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2001f840 con 0x7f2e300a3240 2024-09-16T07:20:56.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:56.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:56.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: dump: 2024-09-16T07:20:56.113 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:56.113 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.113 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.113 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.113 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:56.113 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e277fe700 4 set_mon_vals no callback set 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 10 monclient: _renew_subs 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 1 -- 172.21.15.37:0/3853055398 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e30124830 con 0x7f2e300a3240 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f2e20025030 con 0x7f2e300a3240 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2e14035630 0x7f2e14037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f2e2004d050 con 0x7f2e300a3240 2024-09-16T07:20:56.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:56.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e301383a0 0x7f2e3002b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e36543700 1 -- 172.21.15.37:0/3853055398 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e30067530 con 0x7f2e301383a0 2024-09-16T07:20:56.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35541700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2e14035630 0x7f2e14037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35541700 10 monclient: get_auth_request con 0x7f2e14035630 auth_method 0 2024-09-16T07:20:56.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e301383a0 0x7f2e3002b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 10 monclient: get_auth_request con 0x7f2e301383a0 auth_method 0 2024-09-16T07:20:56.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:56.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:56.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35541700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2e14035630 0x7f2e14037af0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c000f60 tx=0x7f2e2c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 10 monclient(hunting): handle_auth_done global_id 5041 payload 824 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 1 -- 172.21.15.37:0/3853055398 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e300a0d20 con 0x7f2e301383a0 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e35d42700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e301383a0 0x7f2e3002b6a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2e28004750 tx=0x7f2e28007610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e280048f0 con 0x7f2e301383a0 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.092+0000 7f2e25ffb700 10 monclient: dump: 2024-09-16T07:20:56.116 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.096+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f2e2800f050 con 0x7f2e301383a0 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.096+0000 7f2e25ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:56.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.096+0000 7f2e25ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:56.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.096+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e301383a0 msgr2=0x7f2e3002b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.096+0000 7f2e25ffb700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e301383a0 0x7f2e3002b6a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2e28004750 tx=0x7f2e28007610 comp rx=0 tx=0).stop 2024-09-16T07:20:56.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.096+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e301383a0 msgr2=0x7f2e3002b6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:56.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.096+0000 7f2e25ffb700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e301383a0 0x7f2e3002b6a0 secure :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2e28004750 tx=0x7f2e28007610 comp rx=0 tx=0).stop 2024-09-16T07:20:56.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e36543700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:20:56.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e36543700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e3002bbe0 0x7f2e3002bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e36543700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e36543700 1 -- 172.21.15.37:0/3853055398 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3011d660 con 0x7f2e3002bbe0 2024-09-16T07:20:56.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e35d42700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e3002bbe0 0x7f2e3002bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e35d42700 10 monclient: get_auth_request con 0x7f2e3002bbe0 auth_method 0 2024-09-16T07:20:56.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e35d42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:56.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e35d42700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:56.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e35d42700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:56.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e35d42700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:56.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e35d42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:56.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.104+0000 7f2e35d42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:56.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e35d42700 10 monclient(hunting): handle_auth_done global_id 5044 payload 824 2024-09-16T07:20:56.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e35d42700 1 -- 172.21.15.37:0/3853055398 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2e301c8960 con 0x7f2e3002bbe0 2024-09-16T07:20:56.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e35d42700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e3002bbe0 0x7f2e3002bff0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2e28004750 tx=0x7f2e28004af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:56.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2801e2f0 con 0x7f2e3002bbe0 2024-09-16T07:20:56.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:56.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:20:56.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 10 monclient: dump: 2024-09-16T07:20:56.123 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:56.123 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.123 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.123 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.123 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:56.123 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.123 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:56.123 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:56.124 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:56.124 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4787 (secure 0 0 0) 0x7f2e280145f0 con 0x7f2e3002bbe0 2024-09-16T07:20:56.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:20:56.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:56.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e3002bbe0 msgr2=0x7f2e3002bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e3002bbe0 0x7f2e3002bff0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2e28004750 tx=0x7f2e28004af0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 1 -- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e3002bbe0 msgr2=0x7f2e3002bff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:20:56.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e25ffb700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e3002bbe0 0x7f2e3002bff0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 16, 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:56.125 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:56.126 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:56.127 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:20:56.128 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:20:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:20:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:20:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:20:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:20:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:20:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:20:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:20:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:20:56.129 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:20:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:20:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:20:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:20:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:56.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:20:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:20:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:56.132 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:20:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:20:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:20:56.133 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:20:56.134 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.135 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:20:56.136 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:56.137 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:56.138 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:20:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:20:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-09-16T07:20:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:56.139 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:20:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-09-16T07:20:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:20:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:20:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:20:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:20:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:20:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:20:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:20:56.140 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:20:56.140 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:20:56.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 1 -- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2e14035630 msgr2=0x7f2e14037af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2e14035630 0x7f2e14037af0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c000f60 tx=0x7f2e2c009040 comp rx=0 tx=0).stop 2024-09-16T07:20:56.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 10 monclient: shutdown 2024-09-16T07:20:56.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:56.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e35d42700 1 -- 172.21.15.37:0/3853055398 reap_dead start 2024-09-16T07:20:56.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 1 -- 172.21.15.37:0/3853055398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 msgr2=0x7f2e30137e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 0x7f2e30137e60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2e3006b870 tx=0x7f2e20004770 comp rx=0 tx=0).stop 2024-09-16T07:20:56.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 1 -- 172.21.15.37:0/3853055398 shutdown_connections 2024-09-16T07:20:56.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 1 --2- 172.21.15.37:0/3853055398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e300a3240 0x7f2e30137e60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 1 -- 172.21.15.37:0/3853055398 >> 172.21.15.37:0/3853055398 conn(0x7f2e300a3ea0 msgr2=0x7f2e3006e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:56.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 1 -- 172.21.15.37:0/3853055398 shutdown_connections 2024-09-16T07:20:56.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.108+0000 7f2e36543700 1 -- 172.21.15.37:0/3853055398 wait complete. 2024-09-16T07:20:56.143 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:20:56.144 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T07:20:56.144 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T07:20:56.144 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T07:20:56.144 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T07:20:56.144 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T07:20:56.144 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:20:56.190 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T07:20:56.190 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T07:20:56.190 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:20:56.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient: get_monmap_and_config 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient: build_initial_monmap 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient: monmap: 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:56.312314+0000 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:56.312314+0000 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:56.311 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:56.312 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:56.312 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:56.312 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 1 Processor -- start 2024-09-16T07:20:56.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 1 -- start start 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient: init 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48f81356c0 addr v1:172.21.15.143:6789/0 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f48f80a0c50 0x7f48f809dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient(hunting): picked mon.noname-a con 0x7f48f80a0c50 addr v2:172.21.15.37:3300/0 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f806da40 0x7f48f813c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient(hunting): picked mon.noname-b con 0x7f48f806da40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f809e9e0 con 0x7f48f81356c0 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f809ecc0 con 0x7f48f80a0c50 2024-09-16T07:20:56.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f81361b0 con 0x7f48f806da40 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48ffc40700 10 monclient(hunting): authenticate will time out at 976.799109s 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fe1dd700 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f806da40 msgr2=0x7f48f813c550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fe1dd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f806da40 0x7f48f813c550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f48f80a0c50 0x7f48f809dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f48f80a0c50 0x7f48f809dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55992/0 (socket says 172.21.15.37:55992) 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 1 -- 172.21.15.37:0/3206906489 learned_addr learned my addr 172.21.15.37:0/3206906489 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 10 monclient(hunting): get_auth_request con 0x7f48f80a0c50 auth_method 0 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 10 monclient(hunting): handle_auth_done global_id 34191 payload 274 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 1 -- 172.21.15.37:0/3206906489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f806da40 msgr2=0x7f48f813c550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 1 --2- 172.21.15.37:0/3206906489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f806da40 0x7f48f813c550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 1 -- 172.21.15.37:0/3206906489 >> v1:172.21.15.143:6789/0 conn(0x7f48f81356c0 legacy=0x7f48f8135aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 10 monclient: _finish_hunting 0 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 1 monclient: found mon.noname-a 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.308+0000 7f48fd1db700 1 -- 172.21.15.37:0/3206906489 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f80a0720 con 0x7f48f80a0c50 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fd1db700 1 --2- 172.21.15.37:0/3206906489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f80a0c50 0x7f48f809dfa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f48e8009940 tx=0x7f48e8009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2199cc316c8963 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:56.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 1 -- 172.21.15.37:0/3206906489 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e800ba20 con 0x7f48f80a0c50 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 10 monclient: dump: 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 10 monclient: _finish_auth 0 2024-09-16T07:20:56.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 10 monclient: _check_auth_tickets 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 1 -- 172.21.15.37:0/3206906489 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e8009e90 con 0x7f48f80a0c50 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 5 monclient: authenticate success, global_id 34191 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 1 -- 172.21.15.37:0/3206906489 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e800bda0 con 0x7f48f80a0c50 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 4 set_mon_vals no callback set 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48fc9da700 10 monclient: dump: 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.319 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.320 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.320 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:56.320 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.320 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:56.320 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:56.320 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:56.320 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 10 monclient: shutdown 2024-09-16T07:20:56.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:56.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 1 -- 172.21.15.37:0/3206906489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f80a0c50 msgr2=0x7f48f809dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 1 --2- 172.21.15.37:0/3206906489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f80a0c50 0x7f48f809dfa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f48e8009940 tx=0x7f48e8009c50 comp rx=0 tx=0).stop 2024-09-16T07:20:56.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 1 -- 172.21.15.37:0/3206906489 shutdown_connections 2024-09-16T07:20:56.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 1 --2- 172.21.15.37:0/3206906489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f806da40 0x7f48f813c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 1 --2- 172.21.15.37:0/3206906489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f80a0c50 0x7f48f809dfa0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 1 -- 172.21.15.37:0/3206906489 >> 172.21.15.37:0/3206906489 conn(0x7f48f80a1610 msgr2=0x7f48f80a1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 1 -- 172.21.15.37:0/3206906489 shutdown_connections 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 1 -- 172.21.15.37:0/3206906489 wait complete. 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 10 monclient: build_initial_monmap 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.312+0000 7f48ffc40700 10 monclient: monmap: 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:56.325 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 1 Processor -- start 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 1 -- start start 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient: init 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f806da40 0x7f48f81ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient(hunting): picked mon.noname-a con 0x7f48f806da40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80a0c50 0x7f48f8134660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient(hunting): picked mon.noname-b con 0x7f48f80a0c50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f81356c0 0x7f48f811b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient(hunting): picked mon.noname-c con 0x7f48f81356c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f811ee10 con 0x7f48f806da40 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f811e750 con 0x7f48f80a0c50 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f81cb040 con 0x7f48f81356c0 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient(hunting): authenticate will time out at 976.805174s 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fe1dd700 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f81356c0 msgr2=0x7f48f811b160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fe1dd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f81356c0 0x7f48f811b160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f806da40 0x7f48f81ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f806da40 0x7f48f81ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56000/0 (socket says 172.21.15.37:56000) 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 1 -- 172.21.15.37:0/3823600273 learned_addr learned my addr 172.21.15.37:0/3823600273 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 10 monclient(hunting): get_auth_request con 0x7f48f806da40 auth_method 0 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd1db700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80a0c50 0x7f48f8134660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd1db700 10 monclient(hunting): get_auth_request con 0x7f48f80a0c50 auth_method 0 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd1db700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd1db700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:56.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd1db700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd1db700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd1db700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd1db700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 10 monclient(hunting): handle_auth_done global_id 34194 payload 824 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 1 -- 172.21.15.37:0/3823600273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f81356c0 msgr2=0x7f48f811b160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f81356c0 0x7f48f811b160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 1 -- 172.21.15.37:0/3823600273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80a0c50 msgr2=0x7f48f8134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80a0c50 0x7f48f8134660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 10 monclient: _finish_hunting 0 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 1 monclient: found mon.noname-a 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:56.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 1 -- 172.21.15.37:0/3823600273 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e8009570 con 0x7f48f806da40 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48fd9dc700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f806da40 0x7f48f81ca120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f48f400c4e0 tx=0x7f48f400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 1 -- 172.21.15.37:0/3823600273 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f40052d0 con 0x7f48f806da40 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 10 monclient: dump: 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.330 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 10 monclient: _finish_auth 0 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 5 monclient: authenticate success, global_id 34194 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 1 -- 172.21.15.37:0/3823600273 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f4005430 con 0x7f48f806da40 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient: _renew_subs 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 1 -- 172.21.15.37:0/3823600273 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f40055a0 con 0x7f48f806da40 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:56.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 1 -- 172.21.15.37:0/3823600273 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f8072760 con 0x7f48f806da40 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ef7fe700 4 set_mon_vals no callback set 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ee7fc700 10 monclient: dump: 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:56.332 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient: _renew_subs 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.316+0000 7f48ffc40700 1 -- 172.21.15.37:0/3823600273 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f811cc20 con 0x7f48f806da40 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.320+0000 7f48ee7fc700 1 -- 172.21.15.37:0/3823600273 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f48f4003900 con 0x7f48f806da40 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.320+0000 7f48ffc40700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.320+0000 7f48ffc40700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.320+0000 7f48ffc40700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.320+0000 7f48ffc40700 1 -- 172.21.15.37:0/3823600273 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f809dfa0 con 0x7f48f806da40 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.320+0000 7f48ee7fc700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f48dc0355b0 0x7f48dc037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.320+0000 7f48ee7fc700 1 -- 172.21.15.37:0/3823600273 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f48f4046dd0 con 0x7f48f806da40 2024-09-16T07:20:56.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.320+0000 7f48fd1db700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f48dc0355b0 0x7f48dc037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:56.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.320+0000 7f48fd1db700 10 monclient: get_auth_request con 0x7f48dc0355b0 auth_method 0 2024-09-16T07:20:56.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.320+0000 7f48fd1db700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f48dc0355b0 0x7f48dc037a70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f48e8009890 tx=0x7f48e8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:56.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.328+0000 7f48ee7fc700 1 -- 172.21.15.37:0/3823600273 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f48f4018260 con 0x7f48f806da40 2024-09-16T07:20:56.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.328+0000 7f48ee7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:56.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.328+0000 7f48ee7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:56.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.568+0000 7f48ffc40700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:20:56.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.568+0000 7f48ffc40700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:20:56.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.568+0000 7f48ffc40700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:56.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.568+0000 7f48ffc40700 1 -- 172.21.15.37:0/3823600273 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f48f8069740 con 0x7f48f806da40 2024-09-16T07:20:56.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.568+0000 7f48ee7fc700 1 -- 172.21.15.37:0/3823600273 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f48f4017020 con 0x7f48f806da40 2024-09-16T07:20:56.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.568+0000 7f48ee7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:20:56.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.568+0000 7f48ee7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:20:56.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 -- 172.21.15.37:0/3823600273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f48dc0355b0 msgr2=0x7f48dc037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f48dc0355b0 0x7f48dc037a70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f48e8009890 tx=0x7f48e8008030 comp rx=0 tx=0).stop 2024-09-16T07:20:56.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 10 monclient: shutdown 2024-09-16T07:20:56.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:56.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 -- 172.21.15.37:0/3823600273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f806da40 msgr2=0x7f48f81ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:56.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f806da40 0x7f48f81ca120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f48f400c4e0 tx=0x7f48f400c7f0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 -- 172.21.15.37:0/3823600273 shutdown_connections 2024-09-16T07:20:56.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f48dc0355b0 0x7f48dc037a70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48f81356c0 0x7f48f811b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48f80a0c50 0x7f48f8134660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 --2- 172.21.15.37:0/3823600273 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48f806da40 0x7f48f81ca120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:56.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 -- 172.21.15.37:0/3823600273 >> 172.21.15.37:0/3823600273 conn(0x7f48f80a1610 msgr2=0x7f48f80a3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:56.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 -- 172.21.15.37:0/3823600273 shutdown_connections 2024-09-16T07:20:56.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:56.572+0000 7f48ffc40700 1 -- 172.21.15.37:0/3823600273 wait complete. 2024-09-16T07:20:59.587 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:20:59.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.699+0000 7f5173203700 10 monclient: get_monmap_and_config 2024-09-16T07:20:59.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.699+0000 7f5173203700 10 monclient: build_initial_monmap 2024-09-16T07:20:59.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.699+0000 7f5173203700 10 monclient: monmap: 2024-09-16T07:20:59.705 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:20:59.706774+0000 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:20:59.706774+0000 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:20:59.706 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:59.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 Processor -- start 2024-09-16T07:20:59.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 -- start start 2024-09-16T07:20:59.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient: init 2024-09-16T07:20:59.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:59.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T07:20:59.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 0x7f516c0a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:59.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient(hunting): picked mon.noname-b con 0x7f516c0a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:59.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f516c0a1270 0x7f516c0a2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:59.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient(hunting): picked mon.noname-a con 0x7f516c0a1270 addr v2:172.21.15.37:3300/0 2024-09-16T07:20:59.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f516c0a2b80 0x7f516c0a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:59.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient(hunting): picked mon.noname-c con 0x7f516c0a2b80 addr v2:172.21.15.143:3300/0 2024-09-16T07:20:59.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:59.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c06cf50 con 0x7f516c0a1270 2024-09-16T07:20:59.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:59.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c09f810 con 0x7f516c0a2b80 2024-09-16T07:20:59.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:59.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c09f250 con 0x7f516c0a0e90 2024-09-16T07:20:59.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:59.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient(hunting): authenticate will time out at 980.193186s 2024-09-16T07:20:59.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5172201700 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 msgr2=0x7f516c0a04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:20:59.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5172201700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 0x7f516c0a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:20:59.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f516c0a1270 0x7f516c0a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:59.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f516c0a1270 0x7f516c0a2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56016/0 (socket says 172.21.15.37:56016) 2024-09-16T07:20:59.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 1 -- 172.21.15.37:0/3379832102 learned_addr learned my addr 172.21.15.37:0/3379832102 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:59.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 10 monclient(hunting): get_auth_request con 0x7f516c0a1270 auth_method 0 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5172a02700 1 --2- 172.21.15.37:0/3379832102 >> v2:172.21.15.143:3300/0 conn(0x7f516c0a2b80 0x7f516c0a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5172a02700 10 monclient(hunting): get_auth_request con 0x7f516c0a2b80 auth_method 0 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5172a02700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5172a02700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5172a02700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:59.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 10 monclient(hunting): handle_auth_done global_id 34197 payload 274 2024-09-16T07:20:59.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 1 -- 172.21.15.37:0/3379832102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 msgr2=0x7f516c0a04d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:20:59.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 1 --2- 172.21.15.37:0/3379832102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 0x7f516c0a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 1 -- 172.21.15.37:0/3379832102 >> v2:172.21.15.143:3300/0 conn(0x7f516c0a2b80 msgr2=0x7f516c0a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:20:59.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 1 --2- 172.21.15.37:0/3379832102 >> v2:172.21.15.143:3300/0 conn(0x7f516c0a2b80 0x7f516c0a2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 10 monclient: _finish_hunting 0 2024-09-16T07:20:59.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 1 monclient: found mon.noname-a 2024-09-16T07:20:59.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:59.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:59.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 1 -- 172.21.15.37:0/3379832102 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c09f530 con 0x7f516c0a1270 2024-09-16T07:20:59.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5171a00700 1 --2- 172.21.15.37:0/3379832102 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 0x7f516c0a2640 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f516c067820 tx=0x7f515c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c647e37a1cc4a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 1 -- 172.21.15.37:0/3379832102 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00b8a0 con 0x7f516c0a1270 2024-09-16T07:20:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 10 monclient: dump: 2024-09-16T07:20:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:59.725 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:59.726 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 10 monclient: _finish_auth 0 2024-09-16T07:20:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 10 monclient: _check_auth_tickets 2024-09-16T07:20:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 5 monclient: authenticate success, global_id 34197 2024-09-16T07:20:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 1 -- 172.21.15.37:0/3379832102 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c00ba00 con 0x7f516c0a1270 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient: get_monmap_and_config success 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 1 -- 172.21.15.37:0/3379832102 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00bd90 con 0x7f516c0a1270 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 4 set_mon_vals no callback set 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f51711ff700 10 monclient: dump: 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:59.727 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 10 monclient: shutdown 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 -- 172.21.15.37:0/3379832102 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 msgr2=0x7f516c0a2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 --2- 172.21.15.37:0/3379832102 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 0x7f516c0a2640 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f516c067820 tx=0x7f515c009bf0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 -- 172.21.15.37:0/3379832102 shutdown_connections 2024-09-16T07:20:59.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 --2- 172.21.15.37:0/3379832102 >> v2:172.21.15.143:3300/0 conn(0x7f516c0a2b80 0x7f516c0a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 --2- 172.21.15.37:0/3379832102 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 0x7f516c0a2640 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 --2- 172.21.15.37:0/3379832102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 0x7f516c0a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 -- 172.21.15.37:0/3379832102 >> 172.21.15.37:0/3379832102 conn(0x7f516c0a4740 msgr2=0x7f516c0a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:59.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 -- 172.21.15.37:0/3379832102 shutdown_connections 2024-09-16T07:20:59.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.703+0000 7f5173203700 1 -- 172.21.15.37:0/3379832102 wait complete. 2024-09-16T07:20:59.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient: build_initial_monmap 2024-09-16T07:20:59.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient: monmap: 2024-09-16T07:20:59.730 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:20:59.730 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:20:59.730 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:20:59.730 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:20:59.730 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:20:59.730 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:20:59.731 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:20:59.731 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:20:59.731 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:20:59.731 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:59.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 1 Processor -- start 2024-09-16T07:20:59.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 1 -- start start 2024-09-16T07:20:59.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient: init 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient: _reopen_session rank -1 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 0x7f516c1c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient(hunting): picked mon.noname-c con 0x7f516c0a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 0x7f516c1c3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient(hunting): picked mon.noname-a con 0x7f516c0a1270 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c0a2b80 0x7f516c1c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient(hunting): picked mon.noname-b con 0x7f516c0a2b80 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c1ab030 con 0x7f516c0a1270 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:59.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c09dfb0 con 0x7f516c0a2b80 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient(hunting): start opening mon connection 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5172201700 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 msgr2=0x7f516c1c69e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c06c040 con 0x7f516c0a0e90 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient(hunting): _renew_subs 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5173203700 10 monclient(hunting): authenticate will time out at 980.198643s 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5172201700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 0x7f516c1c69e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 0x7f516c1c3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 0x7f516c1c3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56030/0 (socket says 172.21.15.37:56030) 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 1 -- 172.21.15.37:0/3516218940 learned_addr learned my addr 172.21.15.37:0/3516218940 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 10 monclient(hunting): get_auth_request con 0x7f516c0a1270 auth_method 0 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:59.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5172a02700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c0a2b80 0x7f516c1c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5172a02700 10 monclient(hunting): get_auth_request con 0x7f516c0a2b80 auth_method 0 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5172a02700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5172a02700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5172a02700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5172a02700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5172a02700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5172a02700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:20:59.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 10 monclient(hunting): handle_auth_done global_id 34200 payload 824 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 1 -- 172.21.15.37:0/3516218940 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 msgr2=0x7f516c1c69e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 0x7f516c1c69e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 1 -- 172.21.15.37:0/3516218940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c0a2b80 msgr2=0x7f516c1c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c0a2b80 0x7f516c1c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 10 monclient: _finish_hunting 0 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 1 monclient: found mon.noname-a 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.707+0000 7f5171a00700 1 -- 172.21.15.37:0/3516218940 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c009570 con 0x7f516c0a1270 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5171a00700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 0x7f516c1c3f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f515c00b0e0 tx=0x7f515c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 1 -- 172.21.15.37:0/3516218940 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00be50 con 0x7f516c0a1270 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:59.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 10 monclient: dump: 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:59.736 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 10 monclient: _finish_auth 0 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 5 monclient: authenticate success, global_id 34200 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 1 -- 172.21.15.37:0/3516218940 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c01f070 con 0x7f516c0a1270 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 10 monclient: _renew_subs 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 1 -- 172.21.15.37:0/3516218940 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516c06d780 con 0x7f516c0a1270 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 1 -- 172.21.15.37:0/3516218940 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c003900 con 0x7f516c0a1270 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:20:59.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 10 monclient: dump: 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f51709fe700 4 set_mon_vals no callback set 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 10 monclient: _renew_subs 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:59.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 1 -- 172.21.15.37:0/3516218940 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516c1ab8d0 con 0x7f516c0a1270 2024-09-16T07:20:59.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:59.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:59.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:59.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5173203700 1 -- 172.21.15.37:0/3516218940 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516c1ab1a0 con 0x7f516c0a1270 2024-09-16T07:20:59.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 1 -- 172.21.15.37:0/3516218940 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f515c021030 con 0x7f516c0a1270 2024-09-16T07:20:59.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f51500355b0 0x7f5150037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:20:59.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5162ffd700 1 -- 172.21.15.37:0/3516218940 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f515c049010 con 0x7f516c0a1270 2024-09-16T07:20:59.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5172201700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f51500355b0 0x7f5150037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:20:59.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5172201700 10 monclient: get_auth_request con 0x7f51500355b0 auth_method 0 2024-09-16T07:20:59.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.711+0000 7f5172201700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f51500355b0 0x7f5150037a70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5168003a30 tx=0x7f5168009270 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:20:59.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.719+0000 7f5162ffd700 1 -- 172.21.15.37:0/3516218940 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f515c02c9d0 con 0x7f516c0a1270 2024-09-16T07:20:59.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.719+0000 7f5162ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:20:59.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.719+0000 7f5162ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:20:59.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.951+0000 7f5173203700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:20:59.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.951+0000 7f5173203700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:20:59.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.951+0000 7f5173203700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:20:59.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.951+0000 7f5173203700 1 -- 172.21.15.37:0/3516218940 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f516c1ab480 con 0x7f516c0a1270 2024-09-16T07:20:59.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5162ffd700 1 -- 172.21.15.37:0/3516218940 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f515c015020 con 0x7f516c0a1270 2024-09-16T07:20:59.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5162ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:20:59.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5162ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:20:59.959 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 -- 172.21.15.37:0/3516218940 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f51500355b0 msgr2=0x7f5150037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f51500355b0 0x7f5150037a70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5168003a30 tx=0x7f5168009270 comp rx=0 tx=0).stop 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 10 monclient: shutdown 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 -- 172.21.15.37:0/3516218940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 msgr2=0x7f516c1c3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 0x7f516c1c3f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f515c00b0e0 tx=0x7f515c004770 comp rx=0 tx=0).stop 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 -- 172.21.15.37:0/3516218940 shutdown_connections 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f516c0a2b80 0x7f516c1c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f516c0a1270 0x7f516c1c3f70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f51500355b0 0x7f5150037a70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 --2- 172.21.15.37:0/3516218940 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f516c0a0e90 0x7f516c1c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 -- 172.21.15.37:0/3516218940 >> 172.21.15.37:0/3516218940 conn(0x7f516c0a4740 msgr2=0x7f516c09c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:20:59.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 -- 172.21.15.37:0/3516218940 shutdown_connections 2024-09-16T07:20:59.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:20:59.955+0000 7f5173203700 1 -- 172.21.15.37:0/3516218940 wait complete. 2024-09-16T07:21:02.974 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:21:03.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.087+0000 7fbc03fff700 10 monclient: get_monmap_and_config 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.087+0000 7fbc03fff700 10 monclient: build_initial_monmap 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.087+0000 7fbc03fff700 10 monclient: monmap: 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:21:03.095201+0000 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:21:03.095201+0000 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:21:03.094 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:03.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 Processor -- start 2024-09-16T07:21:03.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 -- start start 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient: init 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbc040a0a80 0x7fbc0409d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient(hunting): picked mon.noname-a con 0x7fbc040a0a80 addr v2:172.21.15.37:3300/0 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbc0409a1c0 0x7fbc040a2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient(hunting): picked mon.noname-c con 0x7fbc0409a1c0 addr v2:172.21.15.143:3300/0 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc0409dc20 addr v1:172.21.15.143:6789/0 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0409a640 con 0x7fbc0409dc20 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc040a0e60 con 0x7fbc040a0a80 2024-09-16T07:21:03.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc04131c30 con 0x7fbc0409a1c0 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient(hunting): authenticate will time out at 983.581057s 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbc040a0a80 0x7fbc0409d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbc040a0a80 0x7fbc0409d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58980/0 (socket says 172.21.15.37:58980) 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 1 -- 172.21.15.37:0/1140279789 learned_addr learned my addr 172.21.15.37:0/1140279789 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 10 monclient(hunting): get_auth_request con 0x7fbc040a0a80 auth_method 0 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbbfa7fc700 1 --2- 172.21.15.37:0/1140279789 >> v2:172.21.15.143:3300/0 conn(0x7fbc0409a1c0 0x7fbc040a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:03.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbbfa7fc700 10 monclient(hunting): get_auth_request con 0x7fbc0409a1c0 auth_method 0 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbbfa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbbfa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbbfa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbbfa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbbfa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbbfa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 10 monclient(hunting): handle_auth_done global_id 34203 payload 274 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 1 -- 172.21.15.37:0/1140279789 >> v2:172.21.15.143:3300/0 conn(0x7fbc0409a1c0 msgr2=0x7fbc040a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 1 --2- 172.21.15.37:0/1140279789 >> v2:172.21.15.143:3300/0 conn(0x7fbc0409a1c0 0x7fbc040a2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 1 -- 172.21.15.37:0/1140279789 >> v1:172.21.15.143:6789/0 conn(0x7fbc0409dc20 legacy=0x7fbc040a2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:03.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 10 monclient: _finish_hunting 0 2024-09-16T07:21:03.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 1 monclient: found mon.noname-a 2024-09-16T07:21:03.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:21:03.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:21:03.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 1 -- 172.21.15.37:0/1140279789 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc04131650 con 0x7fbc040a0a80 2024-09-16T07:21:03.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc02ffd700 1 --2- 172.21.15.37:0/1140279789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc040a0a80 0x7fbc0409d6e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbc0409aba0 tx=0x7fbbf4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8154d809cd3f2a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:03.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 1 -- 172.21.15.37:0/1140279789 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4011830 con 0x7fbc040a0a80 2024-09-16T07:21:03.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:03.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:21:03.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 10 monclient: dump: 2024-09-16T07:21:03.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:03.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:03.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:03.103 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:03.103 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:03.103 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:03.103 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:03.103 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:03.103 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:03.104 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:03.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 10 monclient: _finish_auth 0 2024-09-16T07:21:03.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 10 monclient: _check_auth_tickets 2024-09-16T07:21:03.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:21:03.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 5 monclient: authenticate success, global_id 34203 2024-09-16T07:21:03.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:21:03.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 1 -- 172.21.15.37:0/1140279789 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf4011990 con 0x7fbc040a0a80 2024-09-16T07:21:03.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 4 set_mon_vals no callback set 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 1 -- 172.21.15.37:0/1140279789 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4011d20 con 0x7fbc040a0a80 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc027fc700 10 monclient: dump: 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:03.105 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 10 monclient: shutdown 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 -- 172.21.15.37:0/1140279789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc040a0a80 msgr2=0x7fbc0409d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 --2- 172.21.15.37:0/1140279789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc040a0a80 0x7fbc0409d6e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbc0409aba0 tx=0x7fbbf4009bf0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 -- 172.21.15.37:0/1140279789 shutdown_connections 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 --2- 172.21.15.37:0/1140279789 >> v2:172.21.15.143:3300/0 conn(0x7fbc0409a1c0 0x7fbc040a2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 --2- 172.21.15.37:0/1140279789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc040a0a80 0x7fbc0409d6e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 -- 172.21.15.37:0/1140279789 >> 172.21.15.37:0/1140279789 conn(0x7fbc040a3ea0 msgr2=0x7fbc040a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 -- 172.21.15.37:0/1140279789 shutdown_connections 2024-09-16T07:21:03.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.091+0000 7fbc03fff700 1 -- 172.21.15.37:0/1140279789 wait complete. 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient: build_initial_monmap 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient: monmap: 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 1 Processor -- start 2024-09-16T07:21:03.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 1 -- start start 2024-09-16T07:21:03.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient: init 2024-09-16T07:21:03.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:21:03.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:21:03.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc0409a1c0 0x7fbc04130d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:03.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient(hunting): picked mon.noname-b con 0x7fbc0409a1c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:21:03.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc0409dc20 0x7fbc04139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:03.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient(hunting): picked mon.noname-a con 0x7fbc0409dc20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:21:03.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc040a0a80 0x7fbc0413a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:03.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient(hunting): picked mon.noname-c con 0x7fbc040a0a80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:21:03.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:03.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0411aae0 con 0x7fbc0409dc20 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0411bbf0 con 0x7fbc0409a1c0 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0411c1b0 con 0x7fbc040a0a80 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient(hunting): authenticate will time out at 983.585744s 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc037fe700 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc040a0a80 msgr2=0x7fbc0413a320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc037fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc040a0a80 0x7fbc0413a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc0409dc20 0x7fbc04139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc0409dc20 0x7fbc04139de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58996/0 (socket says 172.21.15.37:58996) 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 1 -- 172.21.15.37:0/574572334 learned_addr learned my addr 172.21.15.37:0/574572334 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 10 monclient(hunting): get_auth_request con 0x7fbc0409dc20 auth_method 0 2024-09-16T07:21:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc02ffd700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc0409a1c0 0x7fbc04130d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc02ffd700 10 monclient(hunting): get_auth_request con 0x7fbc0409a1c0 auth_method 0 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc02ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc02ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc02ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc02ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 10 monclient(hunting): handle_auth_done global_id 34206 payload 824 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 1 -- 172.21.15.37:0/574572334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc040a0a80 msgr2=0x7fbc0413a320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc040a0a80 0x7fbc0413a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 1 -- 172.21.15.37:0/574572334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc0409a1c0 msgr2=0x7fbc04130d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc0409a1c0 0x7fbc04130d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 10 monclient: _finish_hunting 0 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 1 monclient: found mon.noname-a 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 1 -- 172.21.15.37:0/574572334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf4009570 con 0x7fbc0409dc20 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfa7fc700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc0409dc20 0x7fbc04139de0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00e7d0 tx=0x7fbbec00eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 1 -- 172.21.15.37:0/574572334 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec013680 con 0x7fbc0409dc20 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 10 monclient: dump: 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 10 monclient: _finish_auth 0 2024-09-16T07:21:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 10 monclient: _check_auth_tickets 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 5 monclient: authenticate success, global_id 34206 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 1 -- 172.21.15.37:0/574572334 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbec0137e0 con 0x7fbc0409dc20 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 1 -- 172.21.15.37:0/574572334 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec013b70 con 0x7fbc0409dc20 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient: _renew_subs 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc01ffb700 4 set_mon_vals no callback set 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 1 -- 172.21.15.37:0/574572334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0411a0c0 con 0x7fbc0409dc20 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbbfbfff700 10 monclient: dump: 2024-09-16T07:21:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient: _renew_subs 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.095+0000 7fbc03fff700 1 -- 172.21.15.37:0/574572334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc04070960 con 0x7fbc0409dc20 2024-09-16T07:21:03.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.099+0000 7fbc03fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.099+0000 7fbc03fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.099+0000 7fbc03fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.099+0000 7fbc03fff700 1 -- 172.21.15.37:0/574572334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0411bd60 con 0x7fbc0409dc20 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.099+0000 7fbbfbfff700 1 -- 172.21.15.37:0/574572334 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fbbec013cd0 con 0x7fbc0409dc20 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.099+0000 7fbbfbfff700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbbe40355b0 0x7fbbe4037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.099+0000 7fbbfbfff700 1 -- 172.21.15.37:0/574572334 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fbbec047930 con 0x7fbc0409dc20 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.099+0000 7fbc02ffd700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbbe40355b0 0x7fbbe4037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.099+0000 7fbc02ffd700 10 monclient: get_auth_request con 0x7fbbe40355b0 auth_method 0 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.099+0000 7fbc02ffd700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbbe40355b0 0x7fbbe4037a70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4000c00 tx=0x7fbbf4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.107+0000 7fbbfbfff700 1 -- 172.21.15.37:0/574572334 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbbec01bbe0 con 0x7fbc0409dc20 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.107+0000 7fbbfbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:21:03.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.107+0000 7fbbfbfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:21:03.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:21:03.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:21:03.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:03.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 -- 172.21.15.37:0/574572334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbc0406b690 con 0x7fbc0409dc20 2024-09-16T07:21:03.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbbfbfff700 1 -- 172.21.15.37:0/574572334 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7fbbec017020 con 0x7fbc0409dc20 2024-09-16T07:21:03.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbbfbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:21:03.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbbfbfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:21:03.343 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:21:03.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 -- 172.21.15.37:0/574572334 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbbe40355b0 msgr2=0x7fbbe4037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbbe40355b0 0x7fbbe4037a70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4000c00 tx=0x7fbbf4008030 comp rx=0 tx=0).stop 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 10 monclient: shutdown 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 -- 172.21.15.37:0/574572334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc0409dc20 msgr2=0x7fbc04139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc0409dc20 0x7fbc04139de0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00e7d0 tx=0x7fbbec00eae0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 -- 172.21.15.37:0/574572334 shutdown_connections 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbbe40355b0 0x7fbbe4037a70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc040a0a80 0x7fbc0413a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc0409dc20 0x7fbc04139de0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 --2- 172.21.15.37:0/574572334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc0409a1c0 0x7fbc04130d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 -- 172.21.15.37:0/574572334 >> 172.21.15.37:0/574572334 conn(0x7fbc040a3ea0 msgr2=0x7fbc0409f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:21:03.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 -- 172.21.15.37:0/574572334 shutdown_connections 2024-09-16T07:21:03.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:03.339+0000 7fbc03fff700 1 -- 172.21.15.37:0/574572334 wait complete. 2024-09-16T07:21:06.358 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:21:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.475+0000 7f42068cd700 10 monclient: get_monmap_and_config 2024-09-16T07:21:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.475+0000 7f42068cd700 10 monclient: build_initial_monmap 2024-09-16T07:21:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.475+0000 7f42068cd700 10 monclient: monmap: 2024-09-16T07:21:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:21:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:21:06.482593+0000 2024-09-16T07:21:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:21:06.482593+0000 2024-09-16T07:21:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:21:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:21:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:21:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:21:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:21:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:21:06.482 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 1 Processor -- start 2024-09-16T07:21:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 1 -- start start 2024-09-16T07:21:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 10 monclient: init 2024-09-16T07:21:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 10 monclient: _reopen_session rank -1 2024-09-16T07:21:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T07:21:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 10 monclient(hunting): picked mon.noname-c con 0x7f42000a0e90 addr v2:172.21.15.143:3300/0 2024-09-16T07:21:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42000a1270 addr v1:172.21.15.37:6789/0 2024-09-16T07:21:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 10 monclient(hunting): picked mon.noname-a con 0x7f42000a2d50 addr v2:172.21.15.37:3300/0 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000a36c0 con 0x7f42000a1270 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f420006cf50 con 0x7f42000a2d50 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f420009f810 con 0x7f42000a0e90 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 10 monclient(hunting): _renew_subs 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42068cd700 10 monclient(hunting): authenticate will time out at 986.970011s 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42050ca700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f42000a1270 0x7f42000a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44346/0 (socket says 172.21.15.37:44346) 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42050ca700 1 -- 172.21.15.37:0/3524129904 learned_addr learned my addr 172.21.15.37:0/3524129904 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42060cc700 10 monclient(hunting): get_auth_request con 0x7f42000a2d50 auth_method 0 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42060cc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42060cc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42060cc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42058cb700 10 monclient(hunting): get_auth_request con 0x7f42000a0e90 auth_method 0 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42058cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42058cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42058cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42060cc700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 msgr2=0x7f42000a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42060cc700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 msgr2=0x7f42000a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:21:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42058cb700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 msgr2=0x7f42000a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:21:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42058cb700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 msgr2=0x7f42000a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:21:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:21:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:21:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.479+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42060cc700 10 monclient(hunting): get_auth_request con 0x7f42000a2d50 auth_method 0 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42060cc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42060cc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42060cc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42058cb700 10 monclient(hunting): get_auth_request con 0x7f42000a0e90 auth_method 0 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42058cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42058cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42058cb700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:21:06.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42060cc700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 msgr2=0x7f42000a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:21:06.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42060cc700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 msgr2=0x7f42000a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:21:06.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:21:06.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:21:06.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42058cb700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 msgr2=0x7f42000a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:21:06.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42058cb700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 msgr2=0x7f42000a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:21:06.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:21:06.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:06.683+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:21:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42060cc700 10 monclient(hunting): get_auth_request con 0x7f42000a2d50 auth_method 0 2024-09-16T07:21:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42060cc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42060cc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42060cc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:21:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42060cc700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 msgr2=0x7f42000a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:21:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42060cc700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 msgr2=0x7f42000a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:21:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:21:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42058cb700 10 monclient(hunting): get_auth_request con 0x7f42000a0e90 auth_method 0 2024-09-16T07:21:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42058cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42058cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42058cb700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:21:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:21:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42058cb700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 msgr2=0x7f42000a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:21:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42058cb700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 msgr2=0x7f42000a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:21:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:21:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.087+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:21:07.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42060cc700 10 monclient(hunting): get_auth_request con 0x7f42000a2d50 auth_method 0 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42060cc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42060cc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42060cc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42058cb700 10 monclient(hunting): get_auth_request con 0x7f42000a0e90 auth_method 0 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42060cc700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 msgr2=0x7f42000a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42058cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42058cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42060cc700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 msgr2=0x7f42000a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42058cb700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:21:07.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T07:21:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42058cb700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 msgr2=0x7f42000a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:21:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42058cb700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 msgr2=0x7f42000a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:21:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:21:07.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:07.891+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T07:21:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 10 monclient(hunting): tick 2024-09-16T07:21:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 10 monclient(hunting): _check_auth_tickets 2024-09-16T07:21:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 1 monclient(hunting): continuing hunt 2024-09-16T07:21:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T07:21:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 msgr2=0x7f42000a04d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:21:09.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.143:3300/0 conn(0x7f42000a0e90 0x7f42000a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:09.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 1 -- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 msgr2=0x7f42000a3180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:21:09.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 1 --2- 172.21.15.37:0/3524129904 >> v2:172.21.15.37:3300/0 conn(0x7f42000a2d50 0x7f42000a3180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:09.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 1 -- 172.21.15.37:0/3524129904 >> v1:172.21.15.37:6789/0 conn(0x7f42000a1270 legacy=0x7f42000a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:09.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 10 monclient: _add_conns ranks=[1,2] 2024-09-16T07:21:09.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41e4003710 addr v1:172.21.15.143:6789/0 2024-09-16T07:21:09.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41e4007340 0x7f41e4009800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:09.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 10 monclient(hunting): picked mon.noname-b con 0x7f41e4007340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:21:09.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 1 -- 172.21.15.37:0/3524129904 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420009f250 con 0x7f41e4003710 2024-09-16T07:21:09.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:09.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 1 -- 172.21.15.37:0/3524129904 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420006cf50 con 0x7f41e4007340 2024-09-16T07:21:09.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f41f77fe700 10 monclient(hunting): _renew_subs 2024-09-16T07:21:09.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f42050ca700 1 -- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41e4007340 msgr2=0x7f41e4009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:21:09.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.483+0000 7f42050ca700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41e4007340 0x7f41e4009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:21:09.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.687+0000 7f42050ca700 1 -- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41e4007340 msgr2=0x7f41e4009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:21:09.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:09.687+0000 7f42050ca700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41e4007340 0x7f41e4009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:21:10.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.091+0000 7f42050ca700 1 -- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41e4007340 msgr2=0x7f41e4009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:21:10.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.091+0000 7f42050ca700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41e4007340 0x7f41e4009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:21:10.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130459844 0 0) 0x7f420009f250 con 0x7f41e4003710 2024-09-16T07:21:10.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): _negotiate 2024-09-16T07:21:10.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:10.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:10.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): my global_id is 5062 2024-09-16T07:21:10.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec0035d0 con 0x7f41e4003710 2024-09-16T07:21:10.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2238813892 0 0) 0x7f41ec0035d0 con 0x7f41e4003710 2024-09-16T07:21:10.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41e4007340 msgr2=0x7f41e4009800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:21:10.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41e4007340 0x7f41e4009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient: _finish_hunting 0 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 monclient: found mon.noname-c-legacy 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e400afd0 con 0x7f41e4003710 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient: _finish_auth 0 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient: _check_auth_tickets 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 120013200 0 0) 0x7f41fc00c570 con 0x7f41e4003710 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42068cd700 5 monclient: authenticate success, global_id 5062 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42068cd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:21:10.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient: dump: 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T07:21:10.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient: _reopen_session rank -1 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 >> v1:172.21.15.143:6789/0 conn(0x7f41e4003710 legacy=0x7f41e4005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42060cc700 1 -- 172.21.15.37:0/3524129904 reap_dead start 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41ec003a20 0x7f41ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): picked mon.b con 0x7f41ec003a20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f41ec004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): picked mon.a con 0x7f41ec004870 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 0x7f41ec00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): picked mon.c con 0x7f41ec005350 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420006cf50 con 0x7f41ec004870 2024-09-16T07:21:10.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec00c2a0 con 0x7f41ec003a20 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ec00c460 con 0x7f41ec005350 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): _renew_subs 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42048c9700 10 monclient(hunting): _finish_auth 0 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42068cd700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42060cc700 1 -- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 msgr2=0x7f41ec00ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42060cc700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 0x7f41ec00ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f41ec004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient(hunting): get_auth_request con 0x7f41ec004870 auth_method 0 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:10.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42058cb700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41ec003a20 0x7f41ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42058cb700 10 monclient(hunting): get_auth_request con 0x7f41ec003a20 auth_method 0 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42058cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42058cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42058cb700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42058cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42058cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42058cb700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:21:10.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient(hunting): handle_auth_done global_id 5062 payload 293 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 1 -- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 msgr2=0x7f41ec00ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 0x7f41ec00ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 1 -- 172.21.15.37:0/3524129904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41ec003a20 msgr2=0x7f41ec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41ec003a20 0x7f41ec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient: _finish_hunting 0 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 1 monclient: found mon.a 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 1 -- 172.21.15.37:0/3524129904 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec0035d0 con 0x7f41ec004870 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient: _finish_auth 0 2024-09-16T07:21:10.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 10 monclient: _check_auth_tickets 2024-09-16T07:21:10.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:21:10.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.779+0000 7f42050ca700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f41ec004cf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f41f0001120 tx=0x7f41ec0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7698f1af8be46d20 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:10.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0004b50 con 0x7f41ec004870 2024-09-16T07:21:10.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42048c9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:10.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42048c9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:10.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42048c9700 10 monclient: dump: 2024-09-16T07:21:10.795 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:10.795 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:10.795 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f0004cb0 con 0x7f41ec004870 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42048c9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 10 monclient: get_monmap_and_config success 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42048c9700 1 -- 172.21.15.37:0/3524129904 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f00072f0 con 0x7f41ec004870 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42048c9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 4 set_mon_vals no callback set 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42048c9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42048c9700 10 monclient: dump: 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 10 monclient: shutdown 2024-09-16T07:21:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 1 -- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 msgr2=0x7f41ec004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f41ec004cf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f41f0001120 tx=0x7f41ec0058a0 comp rx=0 tx=0).stop 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 1 -- 172.21.15.37:0/3524129904 shutdown_connections 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 0x7f41ec00ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f41ec004cf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 1 --2- 172.21.15.37:0/3524129904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41ec003a20 0x7f41ec003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 1 -- 172.21.15.37:0/3524129904 >> 172.21.15.37:0/3524129904 conn(0x7f42000a4740 msgr2=0x7f42000a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 1 -- 172.21.15.37:0/3524129904 shutdown_connections 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 1 -- 172.21.15.37:0/3524129904 wait complete. 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 10 monclient: build_initial_monmap 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.783+0000 7f42068cd700 10 monclient: monmap: 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:21:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 1 Processor -- start 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 1 -- start start 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient: init 2024-09-16T07:21:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient: _reopen_session rank -1 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41ec003a20 0x7f420006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient(hunting): picked mon.noname-b con 0x7f41ec003a20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f420009be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient(hunting): picked mon.noname-a con 0x7f41ec004870 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 0x7f420009c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient(hunting): picked mon.noname-c con 0x7f41ec005350 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420006f170 con 0x7f41ec004870 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42000713b0 con 0x7f41ec003a20 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42000a4ff0 con 0x7f41ec005350 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient(hunting): _renew_subs 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient(hunting): authenticate will time out at 991.278084s 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42060cc700 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 msgr2=0x7f420009c380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42060cc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 0x7f420009c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f420009be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42058cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41ec003a20 0x7f420006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f420009be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47938/0 (socket says 172.21.15.37:47938) 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 1 -- 172.21.15.37:0/1087319209 learned_addr learned my addr 172.21.15.37:0/1087319209 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 10 monclient(hunting): get_auth_request con 0x7f41ec004870 auth_method 0 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42058cb700 10 monclient(hunting): get_auth_request con 0x7f41ec003a20 auth_method 0 2024-09-16T07:21:10.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42058cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42058cb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42058cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42058cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42058cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42058cb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 10 monclient(hunting): handle_auth_done global_id 34209 payload 824 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 1 -- 172.21.15.37:0/1087319209 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 msgr2=0x7f420009c380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 0x7f420009c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 1 -- 172.21.15.37:0/1087319209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41ec003a20 msgr2=0x7f420006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:10.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41ec003a20 0x7f420006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 10 monclient: _finish_hunting 0 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 1 monclient: found mon.noname-a 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 1 -- 172.21.15.37:0/1087319209 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420006d780 con 0x7f41ec004870 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42050ca700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f420009be40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00c270 tx=0x7f41f0001370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 1 -- 172.21.15.37:0/1087319209 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f00074e0 con 0x7f41ec004870 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 10 monclient: dump: 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:10.803 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 10 monclient: _finish_auth 0 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 10 monclient: _check_auth_tickets 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 1 -- 172.21.15.37:0/1087319209 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f0007640 con 0x7f41ec004870 2024-09-16T07:21:10.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 1 -- 172.21.15.37:0/1087319209 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f00015a0 con 0x7f41ec004870 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f67fc700 10 monclient: dump: 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:10.805 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 5 monclient: authenticate success, global_id 34209 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient: _renew_subs 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f42068cd700 1 -- 172.21.15.37:0/1087319209 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420006ee90 con 0x7f41ec004870 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.787+0000 7f41f7fff700 4 set_mon_vals no callback set 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f42068cd700 10 monclient: _renew_subs 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f42068cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f42068cd700 1 -- 172.21.15.37:0/1087319209 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420006dfa0 con 0x7f41ec004870 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f42068cd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f41f67fc700 1 -- 172.21.15.37:0/1087319209 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f41f000c6f0 con 0x7f41ec004870 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f42068cd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:21:10.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f42068cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:10.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f42068cd700 1 -- 172.21.15.37:0/1087319209 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42000a04d0 con 0x7f41ec004870 2024-09-16T07:21:10.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f41f67fc700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41e4035dc0 0x7f41e4038280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:10.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f41f67fc700 1 -- 172.21.15.37:0/1087319209 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f41f00485e0 con 0x7f41ec004870 2024-09-16T07:21:10.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f42058cb700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41e4035dc0 0x7f41e4038280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:10.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f42058cb700 10 monclient: get_auth_request con 0x7f41e4035dc0 auth_method 0 2024-09-16T07:21:10.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.791+0000 7f42058cb700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41e4035dc0 0x7f41e4038280 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f41fc006d90 tx=0x7f41fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:10.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.803+0000 7f41f67fc700 1 -- 172.21.15.37:0/1087319209 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f41f001e5c0 con 0x7f41ec004870 2024-09-16T07:21:10.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.803+0000 7f41f67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:21:10.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:10.803+0000 7f41f67fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:21:11.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.035+0000 7f42068cd700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:21:11.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:21:11.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:11.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 -- 172.21.15.37:0/1087319209 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f42000a2110 con 0x7f41ec004870 2024-09-16T07:21:11.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f41f67fc700 1 -- 172.21.15.37:0/1087319209 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7f41f001e3e0 con 0x7f41ec004870 2024-09-16T07:21:11.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f41f67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:21:11.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f41f67fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:21:11.043 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:21:11.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 -- 172.21.15.37:0/1087319209 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41e4035dc0 msgr2=0x7f41e4038280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41e4035dc0 0x7f41e4038280 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f41fc006d90 tx=0x7f41fc008040 comp rx=0 tx=0).stop 2024-09-16T07:21:11.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 10 monclient: shutdown 2024-09-16T07:21:11.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:21:11.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 -- 172.21.15.37:0/1087319209 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 msgr2=0x7f420009be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f420009be40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00c270 tx=0x7f41f0001370 comp rx=0 tx=0).stop 2024-09-16T07:21:11.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 -- 172.21.15.37:0/1087319209 shutdown_connections 2024-09-16T07:21:11.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41e4035dc0 0x7f41e4038280 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41ec005350 0x7f420009c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41ec004870 0x7f420009be40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 --2- 172.21.15.37:0/1087319209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41ec003a20 0x7f420006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 -- 172.21.15.37:0/1087319209 >> 172.21.15.37:0/1087319209 conn(0x7f42000a4740 msgr2=0x7f42000a71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:21:11.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 -- 172.21.15.37:0/1087319209 shutdown_connections 2024-09-16T07:21:11.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.039+0000 7f42068cd700 1 -- 172.21.15.37:0/1087319209 wait complete. 2024-09-16T07:21:11.057 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T07:21:11.057 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T07:21:11.057 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T07:21:11.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient: get_monmap_and_config 2024-09-16T07:21:11.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient: build_initial_monmap 2024-09-16T07:21:11.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient: monmap: 2024-09-16T07:21:11.165 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:21:11.165 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.165 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:21:11.165212+0000 2024-09-16T07:21:11.165 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:21:11.165212+0000 2024-09-16T07:21:11.165 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:21:11.165 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.165 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:21:11.165 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:21:11.166 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:21:11.166 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:21:11.166 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:21:11.166 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 1 Processor -- start 2024-09-16T07:21:11.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 1 -- start start 2024-09-16T07:21:11.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient: init 2024-09-16T07:21:11.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:21:11.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T07:21:11.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd3200295a0 0x7fd320023ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient(hunting): picked mon.noname-c con 0x7fd3200295a0 addr v2:172.21.15.143:3300/0 2024-09-16T07:21:11.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd3200248c0 0x7fd320024fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient(hunting): picked mon.noname-a con 0x7fd3200248c0 addr v2:172.21.15.37:3300/0 2024-09-16T07:21:11.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd320024400 0x7fd320025500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient(hunting): picked mon.noname-b con 0x7fd320024400 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:21:11.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd320029980 con 0x7fd3200248c0 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd320026820 con 0x7fd3200295a0 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd320027770 con 0x7fd320024400 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd327fff700 10 monclient(hunting): authenticate will time out at 991.651166s 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd3200248c0 0x7fd320024fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3277fe700 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd320024400 msgr2=0x7fd320025500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd3200248c0 0x7fd320024fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47944/0 (socket says 172.21.15.37:47944) 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3277fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd320024400 0x7fd320025500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:21:11.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 1 -- 172.21.15.37:0/327924146 learned_addr learned my addr 172.21.15.37:0/327924146 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:21:11.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 10 monclient(hunting): get_auth_request con 0x7fd3200248c0 auth_method 0 2024-09-16T07:21:11.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:11.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:11.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:11.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:11.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:11.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:11.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 10 monclient(hunting): handle_auth_done global_id 34212 payload 274 2024-09-16T07:21:11.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd326ffd700 1 --2- 172.21.15.37:0/327924146 >> v2:172.21.15.143:3300/0 conn(0x7fd3200295a0 0x7fd320023ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 1 -- 172.21.15.37:0/327924146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd320024400 msgr2=0x7fd320025500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:21:11.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 1 --2- 172.21.15.37:0/327924146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd320024400 0x7fd320025500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 1 -- 172.21.15.37:0/327924146 >> v2:172.21.15.143:3300/0 conn(0x7fd3200295a0 msgr2=0x7fd320023ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:21:11.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 1 --2- 172.21.15.37:0/327924146 >> v2:172.21.15.143:3300/0 conn(0x7fd3200295a0 0x7fd320023ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 10 monclient: _finish_hunting 0 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 1 monclient: found mon.noname-a 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd3267fc700 1 -- 172.21.15.37:0/327924146 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd320027a50 con 0x7fd3200248c0 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd326ffd700 10 monclient: get_auth_request con 0x7fd3200295a0 auth_method 0 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.159+0000 7fd326ffd700 1 --2- 172.21.15.37:0/327924146 >> v2:172.21.15.143:3300/0 conn(0x7fd3200295a0 0x7fd320023ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 1 --2- 172.21.15.37:0/327924146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 0x7fd320024fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd320029cc0 tx=0x7fd318009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37e5be9eb91c2e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 1 -- 172.21.15.37:0/327924146 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31800b8a0 con 0x7fd3200248c0 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 10 monclient: dump: 2024-09-16T07:21:11.181 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 10 monclient: _finish_auth 0 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 1 -- 172.21.15.37:0/327924146 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31800ba00 con 0x7fd3200248c0 2024-09-16T07:21:11.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 5 monclient: authenticate success, global_id 34212 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 4 set_mon_vals no callback set 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 1 -- 172.21.15.37:0/327924146 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31800bd90 con 0x7fd3200248c0 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd325ffb700 10 monclient: dump: 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.183 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient: shutdown 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 -- 172.21.15.37:0/327924146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 msgr2=0x7fd320024fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 --2- 172.21.15.37:0/327924146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 0x7fd320024fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd320029cc0 tx=0x7fd318009bf0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 -- 172.21.15.37:0/327924146 shutdown_connections 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 --2- 172.21.15.37:0/327924146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd320024400 0x7fd320025500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 --2- 172.21.15.37:0/327924146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 0x7fd320024fc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 --2- 172.21.15.37:0/327924146 >> v2:172.21.15.143:3300/0 conn(0x7fd3200295a0 0x7fd320023ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 -- 172.21.15.37:0/327924146 >> 172.21.15.37:0/327924146 conn(0x7fd320006440 msgr2=0x7fd320006850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 -- 172.21.15.37:0/327924146 shutdown_connections 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 -- 172.21.15.37:0/327924146 wait complete. 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient: build_initial_monmap 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient: monmap: 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:21:11.185 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 Processor -- start 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 -- start start 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient: init 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd320024400 0x7fd3200ba480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient(hunting): picked mon.noname-b con 0x7fd320024400 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200295a0 0x7fd3200ba9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient(hunting): picked mon.noname-a con 0x7fd3200295a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd3201509b0 0x7fd3200a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient(hunting): picked mon.noname-c con 0x7fd3201509b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3201516f0 con 0x7fd3200295a0 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd320023b80 con 0x7fd320024400 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd320023650 con 0x7fd3201509b0 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd327fff700 10 monclient(hunting): authenticate will time out at 991.655176s 2024-09-16T07:21:11.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3277fe700 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd3201509b0 msgr2=0x7fd3200a7fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3277fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd3201509b0 0x7fd3200a7fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200295a0 0x7fd3200ba9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200295a0 0x7fd3200ba9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47954/0 (socket says 172.21.15.37:47954) 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 1 -- 172.21.15.37:0/2562835378 learned_addr learned my addr 172.21.15.37:0/2562835378 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 10 monclient(hunting): get_auth_request con 0x7fd3200295a0 auth_method 0 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd326ffd700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd320024400 0x7fd3200ba480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd326ffd700 10 monclient(hunting): get_auth_request con 0x7fd320024400 auth_method 0 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd326ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd326ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:11.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd326ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd3267fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd326ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.163+0000 7fd326ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 10 monclient(hunting): handle_auth_done global_id 34215 payload 824 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 1 -- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd3201509b0 msgr2=0x7fd3200a7fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd3201509b0 0x7fd3200a7fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 1 -- 172.21.15.37:0/2562835378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd320024400 msgr2=0x7fd3200ba480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd320024400 0x7fd3200ba480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 10 monclient: _finish_hunting 0 2024-09-16T07:21:11.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 1 monclient: found mon.noname-a 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 1 -- 172.21.15.37:0/2562835378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd318009570 con 0x7fd3200295a0 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3267fc700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200295a0 0x7fd3200ba9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd318013000 tx=0x7fd318004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3180038c0 con 0x7fd3200295a0 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: dump: 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.190 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: _finish_auth 0 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 5 monclient: authenticate success, global_id 34215 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 10 monclient: _renew_subs 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd318003a20 con 0x7fd3200295a0 2024-09-16T07:21:11.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 1 -- 172.21.15.37:0/2562835378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3200abe80 con 0x7fd3200295a0 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 10 monclient: _renew_subs 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 1 -- 172.21.15.37:0/2562835378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3200a9b20 con 0x7fd3200295a0 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd324ff9700 4 set_mon_vals no callback set 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318003db0 con 0x7fd3200295a0 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: dump: 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.192 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fd318021070 con 0x7fd3200295a0 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 0x7fd3200a84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd327fff700 1 -- 172.21.15.37:0/2562835378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd320007f80 con 0x7fd3200248c0 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd3140355b0 0x7fd314037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 0x7fd3200a84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 10 monclient: get_auth_request con 0x7fd3200248c0 auth_method 0 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fd318049440 con 0x7fd3200295a0 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 10 monclient(hunting): handle_auth_done global_id 34218 payload 824 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 1 -- 172.21.15.37:0/2562835378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd32002d290 con 0x7fd3200248c0 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3277fe700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd3140355b0 0x7fd314037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd3277fe700 10 monclient: get_auth_request con 0x7fd3140355b0 auth_method 0 2024-09-16T07:21:11.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd326ffd700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 0x7fd3200a84e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00f920 tx=0x7fd31c00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c00da40 con 0x7fd3200248c0 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.167+0000 7fd30f7fe700 10 monclient: dump: 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.195 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.171+0000 7fd3277fe700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd3140355b0 0x7fd314037a70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd328068a10 tx=0x7fd328069050 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:11.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.171+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fd31c00c040 con 0x7fd3200248c0 2024-09-16T07:21:11.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.171+0000 7fd30f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:21:11.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.171+0000 7fd30f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:21:11.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.171+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 msgr2=0x7fd3200a84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.171+0000 7fd30f7fe700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 0x7fd3200a84e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00f920 tx=0x7fd31c00fce0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.171+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 msgr2=0x7fd3200a84e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:21:11.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.171+0000 7fd30f7fe700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200248c0 0x7fd3200a84e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd327fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:21:11.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd327fff700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200a8a20 0x7fd320002970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd327fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd327fff700 1 -- 172.21.15.37:0/2562835378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200ab690 con 0x7fd3200a8a20 2024-09-16T07:21:11.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200a8a20 0x7fd320002970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 10 monclient: get_auth_request con 0x7fd3200a8a20 auth_method 0 2024-09-16T07:21:11.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:11.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 10 monclient(hunting): handle_auth_done global_id 34221 payload 824 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 1 -- 172.21.15.37:0/2562835378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd3200bbcc0 con 0x7fd3200a8a20 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd326ffd700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200a8a20 0x7fd320002970 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00f920 tx=0x7fd31c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c0064e0 con 0x7fd3200a8a20 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd30f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd30f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.179+0000 7fd30f7fe700 10 monclient: dump: 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.199 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4608 (secure 0 0 0) 0x7fd31c024070 con 0x7fd3200a8a20 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd30f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd30f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200a8a20 msgr2=0x7fd320002970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd30f7fe700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200a8a20 0x7fd320002970 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00f920 tx=0x7fd31c004770 comp rx=0 tx=0).stop 2024-09-16T07:21:11.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd30f7fe700 1 -- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200a8a20 msgr2=0x7fd320002970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:21:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd30f7fe700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200a8a20 0x7fd320002970 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00f920 tx=0x7fd31c004770 comp rx=0 tx=0).stop 2024-09-16T07:21:11.201 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:21:11.201 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:21:11.201 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:21:11.201 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:21:11.201 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 20, 2024-09-16T07:21:11.201 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:21:11.201 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:21:11.201 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:21:11.202 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:21:11.203 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:21:11.204 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:21:11.205 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:21:11.206 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.207 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:21:11.208 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:21:11.209 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:21:11.210 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.211 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:21:11.212 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:21:11.213 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:21:11.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 1 -- 172.21.15.37:0/2562835378 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd3140355b0 msgr2=0x7fd314037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd3140355b0 0x7fd314037a70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd328068a10 tx=0x7fd328069050 comp rx=0 tx=0).stop 2024-09-16T07:21:11.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 10 monclient: shutdown 2024-09-16T07:21:11.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:21:11.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 1 -- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200295a0 msgr2=0x7fd3200ba9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200295a0 0x7fd3200ba9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd318013000 tx=0x7fd318004770 comp rx=0 tx=0).stop 2024-09-16T07:21:11.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd3277fe700 1 -- 172.21.15.37:0/2562835378 reap_dead start 2024-09-16T07:21:11.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 1 -- 172.21.15.37:0/2562835378 shutdown_connections 2024-09-16T07:21:11.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 1 --2- 172.21.15.37:0/2562835378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3200295a0 0x7fd3200ba9c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 1 -- 172.21.15.37:0/2562835378 >> 172.21.15.37:0/2562835378 conn(0x7fd320006440 msgr2=0x7fd320021ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:21:11.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 1 -- 172.21.15.37:0/2562835378 shutdown_connections 2024-09-16T07:21:11.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.183+0000 7fd327fff700 1 -- 172.21.15.37:0/2562835378 wait complete. 2024-09-16T07:21:11.216 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T07:21:11.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T07:21:11.216 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T07:21:11.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient: get_monmap_and_config 2024-09-16T07:21:11.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient: build_initial_monmap 2024-09-16T07:21:11.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient: monmap: 2024-09-16T07:21:11.332 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:21:11.332 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.332 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:21:11.332577+0000 2024-09-16T07:21:11.332 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:21:11.332577+0000 2024-09-16T07:21:11.332 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:21:11.332 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.332 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:21:11.332 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:21:11.332 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:21:11.332 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:21:11.333 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:21:11.333 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 1 Processor -- start 2024-09-16T07:21:11.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 1 -- start start 2024-09-16T07:21:11.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient: init 2024-09-16T07:21:11.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:21:11.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T07:21:11.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff00c0b63f0 addr v1:172.21.15.143:6789/0 2024-09-16T07:21:11.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff00c0234f0 0x7ff00c0be850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient(hunting): picked mon.noname-a con 0x7ff00c0234f0 addr v2:172.21.15.37:3300/0 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff00c0bed90 addr v1:172.21.15.37:6789/0 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c0b6030 con 0x7ff00c0bed90 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00c023940 con 0x7ff00c0b63f0 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00c020000 con 0x7ff00c0234f0 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient(hunting): authenticate will time out at 991.818543s 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff00c0234f0 0x7ff00c0be850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0137fe700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff00c0bed90 0x7ff00c0c1180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41434/0 (socket says 172.21.15.37:41434) 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff00c0234f0 0x7ff00c0be850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47974/0 (socket says 172.21.15.37:47974) 2024-09-16T07:21:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0137fe700 1 -- 172.21.15.37:0/4291111944 learned_addr learned my addr 172.21.15.37:0/4291111944 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 10 monclient(hunting): get_auth_request con 0x7ff00c0234f0 auth_method 0 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 10 monclient(hunting): handle_auth_done global_id 34224 payload 274 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 1 -- 172.21.15.37:0/4291111944 >> v1:172.21.15.143:6789/0 conn(0x7ff00c0b63f0 legacy=0x7ff00c0b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 1 -- 172.21.15.37:0/4291111944 >> v1:172.21.15.37:6789/0 conn(0x7ff00c0bed90 legacy=0x7ff00c0c1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 10 monclient: _finish_hunting 0 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 1 monclient: found mon.noname-a 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 1 -- 172.21.15.37:0/4291111944 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c01fce0 con 0x7ff00c0234f0 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 1 -- 172.21.15.37:0/4291111944 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182378073 0 0) 0x7ff00c0b6030 con 0x7ff00c0bed90 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff0127fc700 1 --2- 172.21.15.37:0/4291111944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff00c0234f0 0x7ff00c0be850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feffc009880 tx=0x7feffc009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83433ee35ad7cb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 1 -- 172.21.15.37:0/4291111944 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc004230 con 0x7ff00c0234f0 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:21:11.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 10 monclient: dump: 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 10 monclient: _finish_auth 0 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 5 monclient: authenticate success, global_id 34224 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 1 -- 172.21.15.37:0/4291111944 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feffc004390 con 0x7ff00c0234f0 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff013fff700 4 set_mon_vals no callback set 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 1 -- 172.21.15.37:0/4291111944 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc0036c0 con 0x7ff00c0234f0 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.327+0000 7ff011ffb700 10 monclient: dump: 2024-09-16T07:21:11.347 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient: shutdown 2024-09-16T07:21:11.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- 172.21.15.37:0/4291111944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff00c0234f0 msgr2=0x7ff00c0be850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 --2- 172.21.15.37:0/4291111944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff00c0234f0 0x7ff00c0be850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feffc009880 tx=0x7feffc009b90 comp rx=0 tx=0).stop 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- 172.21.15.37:0/4291111944 shutdown_connections 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 --2- 172.21.15.37:0/4291111944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff00c0234f0 0x7ff00c0be850 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- 172.21.15.37:0/4291111944 >> 172.21.15.37:0/4291111944 conn(0x7ff00c0064b0 msgr2=0x7ff00c0068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- 172.21.15.37:0/4291111944 shutdown_connections 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- 172.21.15.37:0/4291111944 wait complete. 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient: build_initial_monmap 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient: monmap: 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:21:11.349 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 Processor -- start 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- start start 2024-09-16T07:21:11.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient: init 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c0234f0 0x7ff00c01c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient(hunting): picked mon.noname-b con 0x7ff00c0234f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff00c0b63f0 0x7ff00c0b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient(hunting): picked mon.noname-a con 0x7ff00c0b63f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff00c0bed90 0x7ff00c0b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient(hunting): picked mon.noname-c con 0x7ff00c0bed90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c0a2d10 con 0x7ff00c0b63f0 2024-09-16T07:21:11.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c01cd70 con 0x7ff00c0234f0 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c020870 con 0x7ff00c0bed90 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient(hunting): authenticate will time out at 991.821601s 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0137fe700 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff00c0bed90 msgr2=0x7ff00c0b5800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0137fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff00c0bed90 0x7ff00c0b5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff00c0b63f0 0x7ff00c0b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff00c0b63f0 0x7ff00c0b52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47978/0 (socket says 172.21.15.37:47978) 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 1 -- 172.21.15.37:0/3950799612 learned_addr learned my addr 172.21.15.37:0/3950799612 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:21:11.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 10 monclient(hunting): get_auth_request con 0x7ff00c0b63f0 auth_method 0 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff012ffd700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c0234f0 0x7ff00c01c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff012ffd700 10 monclient(hunting): get_auth_request con 0x7ff00c0234f0 auth_method 0 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff012ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff012ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:11.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff012ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 10 monclient(hunting): handle_auth_done global_id 34230 payload 824 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 1 -- 172.21.15.37:0/3950799612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff00c0bed90 msgr2=0x7ff00c0b5800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff00c0bed90 0x7ff00c0b5800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 1 -- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c0234f0 msgr2=0x7ff00c01c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c0234f0 0x7ff00c01c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 10 monclient: _finish_hunting 0 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 1 monclient: found mon.noname-a 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 1 -- 172.21.15.37:0/3950799612 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc0094b0 con 0x7ff00c0b63f0 2024-09-16T07:21:11.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff0127fc700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff00c0b63f0 0x7ff00c0b52c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feffc0090b0 tx=0x7feffc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc006ea0 con 0x7ff00c0b63f0 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 10 monclient: dump: 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.355 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 10 monclient: _finish_auth 0 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feffc011a40 con 0x7ff00c0b63f0 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 5 monclient: authenticate success, global_id 34230 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient: _renew_subs 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- 172.21.15.37:0/3950799612 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00c0a1940 con 0x7ff00c0b63f0 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc011db0 con 0x7ff00c0b63f0 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff00b7fe700 10 monclient: dump: 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient: _renew_subs 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 1 -- 172.21.15.37:0/3950799612 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00c0a1480 con 0x7ff00c0b63f0 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff010ff9700 4 set_mon_vals no callback set 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.331+0000 7ff013fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff013fff700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c1503d0 0x7ff00c1507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff013fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff013fff700 1 -- 172.21.15.37:0/3950799612 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c007f80 con 0x7ff00c1503d0 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7feffc021070 con 0x7ff00c0b63f0 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c1503d0 0x7ff00c1507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 10 monclient: get_auth_request con 0x7ff00c1503d0 auth_method 0 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7feff40355b0 0x7feff4037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4346+0+0 (secure 0 0 0) 0x7feffc049290 con 0x7ff00c0b63f0 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:11.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff0137fe700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7feff40355b0 0x7feff4037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff0137fe700 10 monclient: get_auth_request con 0x7feff40355b0 auth_method 0 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 10 monclient(hunting): handle_auth_done global_id 5074 payload 824 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 1 -- 172.21.15.37:0/3950799612 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff00c0b6ee0 con 0x7ff00c1503d0 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff012ffd700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c1503d0 0x7ff00c1507b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff00400e4e0 tx=0x7ff00400e8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff004006a90 con 0x7ff00c1503d0 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff0137fe700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7feff40355b0 0x7feff4037a70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff00c01ca60 tx=0x7ff0140749d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 10 monclient: dump: 2024-09-16T07:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ff004006bf0 con 0x7ff00c1503d0 2024-09-16T07:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:21:11.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:21:11.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c1503d0 msgr2=0x7ff00c1507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c1503d0 0x7ff00c1507b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff00400e4e0 tx=0x7ff00400e8a0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c1503d0 msgr2=0x7ff00c1507b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:21:11.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.335+0000 7ff00b7fe700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c1503d0 0x7ff00c1507b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff013fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:21:11.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff013fff700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c150cf0 0x7ff00c002970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:21:11.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff013fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:21:11.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff013fff700 1 -- 172.21.15.37:0/3950799612 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00c0225c0 con 0x7ff00c150cf0 2024-09-16T07:21:11.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff012ffd700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c150cf0 0x7ff00c002970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:21:11.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff012ffd700 10 monclient: get_auth_request con 0x7ff00c150cf0 auth_method 0 2024-09-16T07:21:11.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff012ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff012ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff012ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff012ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff012ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff012ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff012ffd700 10 monclient(hunting): handle_auth_done global_id 5077 payload 824 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.347+0000 7ff012ffd700 1 -- 172.21.15.37:0/3950799612 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff00c025760 con 0x7ff00c150cf0 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff012ffd700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c150cf0 0x7ff00c002970 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff00400e4e0 tx=0x7ff004003810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff004007580 con 0x7ff00c150cf0 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:21:11.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 10 monclient: dump: 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:21:11.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4776 (secure 0 0 0) 0x7ff0040076e0 con 0x7ff00c150cf0 2024-09-16T07:21:11.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:21:11.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:21:11.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c150cf0 msgr2=0x7ff00c002970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c150cf0 0x7ff00c002970 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff00400e4e0 tx=0x7ff004003810 comp rx=0 tx=0).stop 2024-09-16T07:21:11.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 1 -- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c150cf0 msgr2=0x7ff00c002970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:21:11.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff00b7fe700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff00c150cf0 0x7ff00c002970 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.367 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:21:11.367 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 20, 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:21:11.368 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:21:11.369 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:21:11.370 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:21:11.371 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:21:11.372 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:21:11.372 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:21:11.372 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:21:11.372 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:21:11.372 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:21:11.372 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:21:11.372 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:21:11.372 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:21:11.372 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.373 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:21:11.374 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:21:11.375 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:21:11.376 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:21:11.377 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:21:11.378 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:21:11.379 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:21:11.380 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:21:11.381 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:21:11.381 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.381 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.381 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:21:11.381 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.381 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:21:11.381 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:21:11.381 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-09-16T07:21:11.381 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.381 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:21:11.382 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:21:11.383 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:21:11.383 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:21:11.383 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:21:11.383 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:21:11.383 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff013fff700 1 -- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7feff40355b0 msgr2=0x7feff4037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff013fff700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7feff40355b0 0x7feff4037a70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff00c01ca60 tx=0x7ff0140749d0 comp rx=0 tx=0).stop 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff013fff700 10 monclient: shutdown 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff013fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff013fff700 1 -- 172.21.15.37:0/3950799612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff00c0b63f0 msgr2=0x7ff00c0b52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff013fff700 1 --2- 172.21.15.37:0/3950799612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff00c0b63f0 0x7ff00c0b52c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feffc0090b0 tx=0x7feffc004770 comp rx=0 tx=0).stop 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff0137fe700 1 -- 172.21.15.37:0/3950799612 reap_dead start 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff0137fe700 1 -- 172.21.15.37:0/3950799612 reap_dead start 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff013fff700 1 -- 172.21.15.37:0/3950799612 shutdown_connections 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff013fff700 1 -- 172.21.15.37:0/3950799612 >> 172.21.15.37:0/3950799612 conn(0x7ff00c0064b0 msgr2=0x7ff00c028040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:21:11.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff013fff700 1 -- 172.21.15.37:0/3950799612 shutdown_connections 2024-09-16T07:21:11.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:21:11.351+0000 7ff013fff700 1 -- 172.21.15.37:0/3950799612 wait complete. 2024-09-16T07:21:11.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T07:21:11.388 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T07:21:12.904 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:12.904 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25721]: starting. 2024-09-16T07:21:12.904 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25721]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25684 2024-09-16T07:21:12.905 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25721]: created object 0... 2024-09-16T07:21:12.905 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25721]: created object 25... 2024-09-16T07:21:12.905 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25721]: created object 49... 2024-09-16T07:21:12.905 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25721]: finishing. 2024-09-16T07:21:12.905 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25721]: shutting down. 2024-09-16T07:21:12.935 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:12.935 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25723]: starting. 2024-09-16T07:21:12.935 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25723]: listing objects. 2024-09-16T07:21:12.935 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25723]: listed object 0... 2024-09-16T07:21:12.936 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25723]: listed object 25... 2024-09-16T07:21:12.936 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25723]: saw 50 objects 2024-09-16T07:21:12.936 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25723]: shutting down. 2024-09-16T07:21:13.763 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:13.763 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25722]: starting. 2024-09-16T07:21:13.763 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25722]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25684 2024-09-16T07:21:13.763 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25722]: shutting down. 2024-09-16T07:21:13.766 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:13.766 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:13.766 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:21:13.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:21:15.774 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25778]: starting. 2024-09-16T07:21:15.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25778]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25777 2024-09-16T07:21:15.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25778]: created object 0... 2024-09-16T07:21:15.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25778]: created object 25... 2024-09-16T07:21:15.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25778]: created object 49... 2024-09-16T07:21:15.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25778]: finishing. 2024-09-16T07:21:15.775 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25778]: shutting down. 2024-09-16T07:21:16.614 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25779]: starting. 2024-09-16T07:21:16.615 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25779]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25777 2024-09-16T07:21:16.615 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25779]: shutting down. 2024-09-16T07:21:18.782 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:18.782 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25814]: starting. 2024-09-16T07:21:18.782 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25814]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25777 2024-09-16T07:21:18.782 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25814]: created object 0... 2024-09-16T07:21:18.782 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25814]: created object 25... 2024-09-16T07:21:18.782 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25814]: created object 49... 2024-09-16T07:21:18.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25814]: finishing. 2024-09-16T07:21:18.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25814]: shutting down. 2024-09-16T07:21:19.223 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:19.425 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25816]: starting. 2024-09-16T07:21:19.425 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25816]: listing objects. 2024-09-16T07:21:19.425 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25816]: listed object 0... 2024-09-16T07:21:19.426 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25816]: listed object 25... 2024-09-16T07:21:19.426 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25816]: saw 50 objects 2024-09-16T07:21:19.426 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25816]: shutting down. 2024-09-16T07:21:19.626 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:19.626 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25815]: starting. 2024-09-16T07:21:19.626 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25815]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25777 2024-09-16T07:21:19.626 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25815]: shutting down. 2024-09-16T07:21:19.628 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:19.629 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:19.629 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:21:19.631 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:21:21.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25869]: starting. 2024-09-16T07:21:21.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25869]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25868 2024-09-16T07:21:21.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25869]: created object 0... 2024-09-16T07:21:21.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25869]: created object 25... 2024-09-16T07:21:21.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25869]: created object 49... 2024-09-16T07:21:21.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25869]: finishing. 2024-09-16T07:21:21.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25869]: shutting down. 2024-09-16T07:21:22.653 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25870]: starting. 2024-09-16T07:21:22.653 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25870]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25868 2024-09-16T07:21:22.653 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25870]: shutting down. 2024-09-16T07:21:24.819 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:24.820 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25905]: starting. 2024-09-16T07:21:24.820 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25905]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25868 2024-09-16T07:21:24.820 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25905]: created object 0... 2024-09-16T07:21:24.820 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25905]: created object 25... 2024-09-16T07:21:24.820 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25905]: created object 49... 2024-09-16T07:21:24.820 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25905]: finishing. 2024-09-16T07:21:24.820 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25905]: shutting down. 2024-09-16T07:21:24.850 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:24.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25907]: starting. 2024-09-16T07:21:24.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25907]: listing objects. 2024-09-16T07:21:24.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25907]: listed object 0... 2024-09-16T07:21:24.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25907]: listed object 25... 2024-09-16T07:21:24.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25907]: saw 50 objects 2024-09-16T07:21:24.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25907]: shutting down. 2024-09-16T07:21:25.617 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:25.617 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25906]: starting. 2024-09-16T07:21:25.617 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25906]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25868 2024-09-16T07:21:25.617 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25906]: shutting down. 2024-09-16T07:21:25.620 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:25.620 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:25.620 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:21:25.622 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:21:27.835 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25960]: starting. 2024-09-16T07:21:27.835 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25960]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25959 2024-09-16T07:21:27.835 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25960]: created object 0... 2024-09-16T07:21:27.835 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25960]: created object 25... 2024-09-16T07:21:27.835 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25960]: created object 49... 2024-09-16T07:21:27.835 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25960]: finishing. 2024-09-16T07:21:27.835 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[25960]: shutting down. 2024-09-16T07:21:28.691 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25961]: starting. 2024-09-16T07:21:28.691 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25961]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25959 2024-09-16T07:21:28.691 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[25961]: shutting down. 2024-09-16T07:21:30.871 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:30.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25996]: starting. 2024-09-16T07:21:30.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25996]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-25959 2024-09-16T07:21:30.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25996]: created object 0... 2024-09-16T07:21:30.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25996]: created object 25... 2024-09-16T07:21:30.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25996]: created object 49... 2024-09-16T07:21:30.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25996]: finishing. 2024-09-16T07:21:30.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[25996]: shutting down. 2024-09-16T07:21:30.904 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:30.904 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25998]: starting. 2024-09-16T07:21:30.904 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25998]: listing objects. 2024-09-16T07:21:30.904 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25998]: listed object 0... 2024-09-16T07:21:30.904 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25998]: listed object 25... 2024-09-16T07:21:30.905 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25998]: saw 50 objects 2024-09-16T07:21:30.905 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[25998]: shutting down. 2024-09-16T07:21:31.730 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:31.730 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25997]: starting. 2024-09-16T07:21:31.730 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25997]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-25959 2024-09-16T07:21:31.730 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[25997]: shutting down. 2024-09-16T07:21:31.732 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:31.733 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:31.733 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:21:31.735 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:21:33.902 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26051]: starting. 2024-09-16T07:21:33.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26051]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26050 2024-09-16T07:21:33.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26051]: created object 0... 2024-09-16T07:21:33.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26051]: created object 25... 2024-09-16T07:21:33.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26051]: created object 49... 2024-09-16T07:21:33.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26051]: finishing. 2024-09-16T07:21:33.903 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26051]: shutting down. 2024-09-16T07:21:34.770 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26052]: starting. 2024-09-16T07:21:34.770 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26052]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26050 2024-09-16T07:21:34.770 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26052]: shutting down. 2024-09-16T07:21:36.789 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:36.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26087]: starting. 2024-09-16T07:21:36.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26087]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26050 2024-09-16T07:21:36.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26087]: created object 0... 2024-09-16T07:21:36.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26087]: created object 25... 2024-09-16T07:21:36.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26087]: created object 49... 2024-09-16T07:21:36.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26087]: finishing. 2024-09-16T07:21:36.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26087]: shutting down. 2024-09-16T07:21:36.820 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:36.820 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26089]: starting. 2024-09-16T07:21:36.820 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26089]: listing objects. 2024-09-16T07:21:36.820 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26089]: listed object 0... 2024-09-16T07:21:36.821 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26089]: listed object 25... 2024-09-16T07:21:36.821 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26089]: saw 50 objects 2024-09-16T07:21:36.821 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26089]: shutting down. 2024-09-16T07:21:37.628 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:37.628 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26088]: starting. 2024-09-16T07:21:37.628 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26088]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26050 2024-09-16T07:21:37.628 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26088]: shutting down. 2024-09-16T07:21:37.631 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:37.631 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:37.631 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:21:37.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:21:39.802 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26142]: starting. 2024-09-16T07:21:39.803 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26142]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26141 2024-09-16T07:21:39.803 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26142]: created object 0... 2024-09-16T07:21:39.803 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26142]: created object 25... 2024-09-16T07:21:39.803 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26142]: created object 49... 2024-09-16T07:21:39.803 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26142]: finishing. 2024-09-16T07:21:39.803 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26142]: shutting down. 2024-09-16T07:21:40.663 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26143]: starting. 2024-09-16T07:21:40.663 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26143]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26141 2024-09-16T07:21:40.663 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26143]: shutting down. 2024-09-16T07:21:42.841 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:42.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26178]: starting. 2024-09-16T07:21:42.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26178]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26141 2024-09-16T07:21:42.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26178]: created object 0... 2024-09-16T07:21:42.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26178]: created object 25... 2024-09-16T07:21:42.842 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26178]: created object 49... 2024-09-16T07:21:42.842 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26178]: finishing. 2024-09-16T07:21:42.842 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26178]: shutting down. 2024-09-16T07:21:42.870 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:42.870 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26180]: starting. 2024-09-16T07:21:42.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26180]: listing objects. 2024-09-16T07:21:42.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26180]: listed object 0... 2024-09-16T07:21:42.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26180]: listed object 25... 2024-09-16T07:21:42.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26180]: saw 50 objects 2024-09-16T07:21:42.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26180]: shutting down. 2024-09-16T07:21:43.699 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:43.699 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26179]: starting. 2024-09-16T07:21:43.699 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26179]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26141 2024-09-16T07:21:43.699 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26179]: shutting down. 2024-09-16T07:21:43.702 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:43.702 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:43.702 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:21:43.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:21:45.788 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26233]: starting. 2024-09-16T07:21:45.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26233]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26232 2024-09-16T07:21:45.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26233]: created object 0... 2024-09-16T07:21:45.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26233]: created object 25... 2024-09-16T07:21:45.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26233]: created object 49... 2024-09-16T07:21:45.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26233]: finishing. 2024-09-16T07:21:45.789 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26233]: shutting down. 2024-09-16T07:21:46.635 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26234]: starting. 2024-09-16T07:21:46.635 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26234]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26232 2024-09-16T07:21:46.635 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26234]: shutting down. 2024-09-16T07:21:48.800 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:48.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26269]: starting. 2024-09-16T07:21:48.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26269]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26232 2024-09-16T07:21:48.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26269]: created object 0... 2024-09-16T07:21:48.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26269]: created object 25... 2024-09-16T07:21:48.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26269]: created object 49... 2024-09-16T07:21:48.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26269]: finishing. 2024-09-16T07:21:48.800 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26269]: shutting down. 2024-09-16T07:21:48.831 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:48.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26271]: starting. 2024-09-16T07:21:48.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26271]: listing objects. 2024-09-16T07:21:48.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26271]: listed object 0... 2024-09-16T07:21:48.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26271]: listed object 25... 2024-09-16T07:21:48.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26271]: saw 50 objects 2024-09-16T07:21:48.831 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26271]: shutting down. 2024-09-16T07:21:49.644 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:49.644 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26270]: starting. 2024-09-16T07:21:49.644 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26270]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26232 2024-09-16T07:21:49.644 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26270]: shutting down. 2024-09-16T07:21:49.647 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:49.647 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:49.647 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:21:49.649 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:21:51.847 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26324]: starting. 2024-09-16T07:21:51.847 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26324]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26323 2024-09-16T07:21:51.847 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26324]: created object 0... 2024-09-16T07:21:51.847 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26324]: created object 25... 2024-09-16T07:21:51.847 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26324]: created object 49... 2024-09-16T07:21:51.847 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26324]: finishing. 2024-09-16T07:21:51.847 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26324]: shutting down. 2024-09-16T07:21:52.699 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26325]: starting. 2024-09-16T07:21:52.699 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26325]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26323 2024-09-16T07:21:52.699 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26325]: shutting down. 2024-09-16T07:21:54.874 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:54.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26360]: starting. 2024-09-16T07:21:54.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26360]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26323 2024-09-16T07:21:54.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26360]: created object 0... 2024-09-16T07:21:54.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26360]: created object 25... 2024-09-16T07:21:54.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26360]: created object 49... 2024-09-16T07:21:54.875 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26360]: finishing. 2024-09-16T07:21:54.875 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26360]: shutting down. 2024-09-16T07:21:54.905 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:54.906 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26362]: starting. 2024-09-16T07:21:54.906 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26362]: listing objects. 2024-09-16T07:21:54.906 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26362]: listed object 0... 2024-09-16T07:21:54.906 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26362]: listed object 25... 2024-09-16T07:21:54.906 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26362]: saw 50 objects 2024-09-16T07:21:54.906 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26362]: shutting down. 2024-09-16T07:21:55.627 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:55.628 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26361]: starting. 2024-09-16T07:21:55.628 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26361]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26323 2024-09-16T07:21:55.628 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26361]: shutting down. 2024-09-16T07:21:55.629 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:55.630 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:21:55.630 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:21:55.632 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:21:57.805 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26415]: starting. 2024-09-16T07:21:57.806 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26415]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26414 2024-09-16T07:21:57.806 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26415]: created object 0... 2024-09-16T07:21:57.806 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26415]: created object 25... 2024-09-16T07:21:57.806 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26415]: created object 49... 2024-09-16T07:21:57.806 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26415]: finishing. 2024-09-16T07:21:57.806 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26415]: shutting down. 2024-09-16T07:21:58.654 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26416]: starting. 2024-09-16T07:21:58.654 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26416]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26414 2024-09-16T07:21:58.654 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26416]: shutting down. 2024-09-16T07:22:00.832 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:00.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26451]: starting. 2024-09-16T07:22:00.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26451]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26414 2024-09-16T07:22:00.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26451]: created object 0... 2024-09-16T07:22:00.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26451]: created object 25... 2024-09-16T07:22:00.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26451]: created object 49... 2024-09-16T07:22:00.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26451]: finishing. 2024-09-16T07:22:00.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26451]: shutting down. 2024-09-16T07:22:00.863 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:00.863 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26453]: starting. 2024-09-16T07:22:00.863 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26453]: listing objects. 2024-09-16T07:22:00.864 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26453]: listed object 0... 2024-09-16T07:22:00.864 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26453]: listed object 25... 2024-09-16T07:22:00.864 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26453]: saw 50 objects 2024-09-16T07:22:00.864 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26453]: shutting down. 2024-09-16T07:22:01.694 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:01.695 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26452]: starting. 2024-09-16T07:22:01.695 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26452]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26414 2024-09-16T07:22:01.695 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26452]: shutting down. 2024-09-16T07:22:01.697 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:01.697 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:01.698 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:22:01.700 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:22:03.856 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26506]: starting. 2024-09-16T07:22:03.856 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26506]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26505 2024-09-16T07:22:03.856 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26506]: created object 0... 2024-09-16T07:22:03.856 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26506]: created object 25... 2024-09-16T07:22:03.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26506]: created object 49... 2024-09-16T07:22:03.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26506]: finishing. 2024-09-16T07:22:03.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26506]: shutting down. 2024-09-16T07:22:04.723 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26507]: starting. 2024-09-16T07:22:04.724 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26507]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26505 2024-09-16T07:22:04.724 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26507]: shutting down. 2024-09-16T07:22:06.793 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:06.793 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26543]: starting. 2024-09-16T07:22:06.793 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26543]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26505 2024-09-16T07:22:06.793 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26543]: created object 0... 2024-09-16T07:22:06.793 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26543]: created object 25... 2024-09-16T07:22:06.793 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26543]: created object 49... 2024-09-16T07:22:06.793 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26543]: finishing. 2024-09-16T07:22:06.793 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26543]: shutting down. 2024-09-16T07:22:06.824 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26545]: starting. 2024-09-16T07:22:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26545]: listing objects. 2024-09-16T07:22:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26545]: listed object 0... 2024-09-16T07:22:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26545]: listed object 25... 2024-09-16T07:22:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26545]: saw 50 objects 2024-09-16T07:22:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26545]: shutting down. 2024-09-16T07:22:07.639 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:07.639 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26544]: starting. 2024-09-16T07:22:07.639 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26544]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26505 2024-09-16T07:22:07.639 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26544]: shutting down. 2024-09-16T07:22:07.641 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:07.641 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:07.641 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:22:07.644 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:22:09.818 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26598]: starting. 2024-09-16T07:22:09.818 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26598]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26597 2024-09-16T07:22:09.818 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26598]: created object 0... 2024-09-16T07:22:09.818 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26598]: created object 25... 2024-09-16T07:22:09.819 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26598]: created object 49... 2024-09-16T07:22:09.819 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26598]: finishing. 2024-09-16T07:22:09.819 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26598]: shutting down. 2024-09-16T07:22:10.676 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26599]: starting. 2024-09-16T07:22:10.676 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26599]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26597 2024-09-16T07:22:10.676 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26599]: shutting down. 2024-09-16T07:22:12.840 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:12.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26634]: starting. 2024-09-16T07:22:12.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26634]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26597 2024-09-16T07:22:12.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26634]: created object 0... 2024-09-16T07:22:12.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26634]: created object 25... 2024-09-16T07:22:12.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26634]: created object 49... 2024-09-16T07:22:12.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26634]: finishing. 2024-09-16T07:22:12.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26634]: shutting down. 2024-09-16T07:22:12.871 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:12.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26636]: starting. 2024-09-16T07:22:12.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26636]: listing objects. 2024-09-16T07:22:12.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26636]: listed object 0... 2024-09-16T07:22:12.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26636]: listed object 25... 2024-09-16T07:22:12.871 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26636]: saw 50 objects 2024-09-16T07:22:12.872 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26636]: shutting down. 2024-09-16T07:22:13.702 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:13.702 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26635]: starting. 2024-09-16T07:22:13.703 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26635]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26597 2024-09-16T07:22:13.703 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26635]: shutting down. 2024-09-16T07:22:13.705 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:13.706 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:13.706 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:22:13.709 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:22:15.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26689]: starting. 2024-09-16T07:22:15.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26689]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26688 2024-09-16T07:22:15.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26689]: created object 0... 2024-09-16T07:22:15.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26689]: created object 25... 2024-09-16T07:22:15.786 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26689]: created object 49... 2024-09-16T07:22:15.787 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26689]: finishing. 2024-09-16T07:22:15.787 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26689]: shutting down. 2024-09-16T07:22:16.647 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26690]: starting. 2024-09-16T07:22:16.647 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26690]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26688 2024-09-16T07:22:16.647 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26690]: shutting down. 2024-09-16T07:22:18.812 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:18.812 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26725]: starting. 2024-09-16T07:22:18.812 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26725]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26688 2024-09-16T07:22:18.813 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26725]: created object 0... 2024-09-16T07:22:18.813 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26725]: created object 25... 2024-09-16T07:22:18.813 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26725]: created object 49... 2024-09-16T07:22:18.813 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26725]: finishing. 2024-09-16T07:22:18.813 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26725]: shutting down. 2024-09-16T07:22:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26727]: starting. 2024-09-16T07:22:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26727]: listing objects. 2024-09-16T07:22:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26727]: listed object 0... 2024-09-16T07:22:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26727]: listed object 25... 2024-09-16T07:22:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26727]: saw 50 objects 2024-09-16T07:22:18.843 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26727]: shutting down. 2024-09-16T07:22:19.678 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:19.678 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26726]: starting. 2024-09-16T07:22:19.678 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26726]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26688 2024-09-16T07:22:19.678 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26726]: shutting down. 2024-09-16T07:22:19.679 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:19.679 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:19.679 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:22:19.681 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:22:21.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26781]: starting. 2024-09-16T07:22:21.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26781]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26780 2024-09-16T07:22:21.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26781]: created object 0... 2024-09-16T07:22:21.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26781]: created object 25... 2024-09-16T07:22:21.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26781]: created object 49... 2024-09-16T07:22:21.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26781]: finishing. 2024-09-16T07:22:21.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26781]: shutting down. 2024-09-16T07:22:22.704 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26782]: starting. 2024-09-16T07:22:22.704 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26782]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26780 2024-09-16T07:22:22.704 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26782]: shutting down. 2024-09-16T07:22:24.878 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:24.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26817]: starting. 2024-09-16T07:22:24.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26817]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26780 2024-09-16T07:22:24.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26817]: created object 0... 2024-09-16T07:22:24.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26817]: created object 25... 2024-09-16T07:22:24.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26817]: created object 49... 2024-09-16T07:22:24.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26817]: finishing. 2024-09-16T07:22:24.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26817]: shutting down. 2024-09-16T07:22:24.910 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:24.910 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26819]: starting. 2024-09-16T07:22:24.910 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26819]: listing objects. 2024-09-16T07:22:24.910 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26819]: listed object 0... 2024-09-16T07:22:24.910 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26819]: listed object 25... 2024-09-16T07:22:24.911 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26819]: saw 50 objects 2024-09-16T07:22:24.911 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26819]: shutting down. 2024-09-16T07:22:25.640 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:25.640 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26818]: starting. 2024-09-16T07:22:25.640 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26818]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26780 2024-09-16T07:22:25.641 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26818]: shutting down. 2024-09-16T07:22:25.643 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:25.643 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:25.643 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:22:25.645 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:22:27.809 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26872]: starting. 2024-09-16T07:22:27.809 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26872]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26871 2024-09-16T07:22:27.809 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26872]: created object 0... 2024-09-16T07:22:27.809 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26872]: created object 25... 2024-09-16T07:22:27.810 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26872]: created object 49... 2024-09-16T07:22:27.810 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26872]: finishing. 2024-09-16T07:22:27.810 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26872]: shutting down. 2024-09-16T07:22:28.671 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26873]: starting. 2024-09-16T07:22:28.671 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26873]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26871 2024-09-16T07:22:28.671 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26873]: shutting down. 2024-09-16T07:22:30.850 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:30.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26908]: starting. 2024-09-16T07:22:30.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26908]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26871 2024-09-16T07:22:30.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26908]: created object 0... 2024-09-16T07:22:30.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26908]: created object 25... 2024-09-16T07:22:30.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26908]: created object 49... 2024-09-16T07:22:30.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26908]: finishing. 2024-09-16T07:22:30.850 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26908]: shutting down. 2024-09-16T07:22:30.882 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:30.883 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26910]: starting. 2024-09-16T07:22:30.883 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26910]: listing objects. 2024-09-16T07:22:30.883 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26910]: listed object 0... 2024-09-16T07:22:30.883 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26910]: listed object 25... 2024-09-16T07:22:30.883 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26910]: saw 50 objects 2024-09-16T07:22:30.883 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[26910]: shutting down. 2024-09-16T07:22:31.695 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:31.696 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26909]: starting. 2024-09-16T07:22:31.696 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26909]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26871 2024-09-16T07:22:31.696 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[26909]: shutting down. 2024-09-16T07:22:31.696 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:31.696 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:31.696 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:22:31.699 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:22:33.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26963]: starting. 2024-09-16T07:22:33.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26963]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26962 2024-09-16T07:22:33.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26963]: created object 0... 2024-09-16T07:22:33.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26963]: created object 25... 2024-09-16T07:22:33.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26963]: created object 49... 2024-09-16T07:22:33.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26963]: finishing. 2024-09-16T07:22:33.879 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[26963]: shutting down. 2024-09-16T07:22:34.726 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26964]: starting. 2024-09-16T07:22:34.726 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26964]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26962 2024-09-16T07:22:34.726 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[26964]: shutting down. 2024-09-16T07:22:36.807 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:36.807 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26999]: starting. 2024-09-16T07:22:36.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26999]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-26962 2024-09-16T07:22:36.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26999]: created object 0... 2024-09-16T07:22:36.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26999]: created object 25... 2024-09-16T07:22:36.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26999]: created object 49... 2024-09-16T07:22:36.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26999]: finishing. 2024-09-16T07:22:36.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[26999]: shutting down. 2024-09-16T07:22:36.836 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:36.836 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27001]: starting. 2024-09-16T07:22:36.837 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27001]: listing objects. 2024-09-16T07:22:36.837 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27001]: listed object 0... 2024-09-16T07:22:36.837 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27001]: listed object 25... 2024-09-16T07:22:36.837 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27001]: saw 50 objects 2024-09-16T07:22:36.837 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27001]: shutting down. 2024-09-16T07:22:37.658 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:37.658 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27000]: starting. 2024-09-16T07:22:37.658 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27000]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-26962 2024-09-16T07:22:37.658 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27000]: shutting down. 2024-09-16T07:22:37.660 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:37.660 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:37.660 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:22:37.663 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:22:40.051 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27054]: starting. 2024-09-16T07:22:40.051 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27054]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27053 2024-09-16T07:22:40.051 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27054]: created object 0... 2024-09-16T07:22:40.052 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27054]: created object 25... 2024-09-16T07:22:40.052 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27054]: created object 49... 2024-09-16T07:22:40.052 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27054]: finishing. 2024-09-16T07:22:40.052 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27054]: shutting down. 2024-09-16T07:22:40.691 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27055]: starting. 2024-09-16T07:22:40.692 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27055]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27053 2024-09-16T07:22:40.692 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27055]: shutting down. 2024-09-16T07:22:41.388 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T07:22:41.388 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T07:22:41.389 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T07:22:41.389 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T07:22:41.392 INFO:tasks.ceph.mon.c:Started 2024-09-16T07:22:41.392 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T07:22:41.392 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:22:41.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.497+0000 7fb05759e700 10 monclient: get_monmap_and_config 2024-09-16T07:22:41.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.497+0000 7fb05759e700 10 monclient: build_initial_monmap 2024-09-16T07:22:41.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.497+0000 7fb05759e700 10 monclient: monmap: 2024-09-16T07:22:41.502 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:41.502 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:41.502 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:41.502228+0000 2024-09-16T07:22:41.502 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:41.502228+0000 2024-09-16T07:22:41.502 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:41.503 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:41.503 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:41.503 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:41.503 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:41.503 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:41.503 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:41.503 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:41.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.497+0000 7fb05759e700 1 Processor -- start 2024-09-16T07:22:41.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.497+0000 7fb05759e700 1 -- start start 2024-09-16T07:22:41.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.497+0000 7fb05759e700 10 monclient: init 2024-09-16T07:22:41.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:41.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb05809cca0 addr v1:172.21.15.143:6789/0 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb05809a0f0 0x7fb058137040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient(hunting): picked mon.noname-c con 0x7fb05809a0f0 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb05809a540 0x7fb058139590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient(hunting): picked mon.noname-a con 0x7fb05809a540 addr v2:172.21.15.37:3300/0 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05809c3e0 con 0x7fb05809cca0 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb05809d080 con 0x7fb05809a540 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb05809c100 con 0x7fb05809a0f0 2024-09-16T07:22:41.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient(hunting): authenticate will time out at 1081.989300s 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb05809a540 0x7fb058139590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb05809a540 0x7fb058139590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51962/0 (socket says 172.21.15.37:51962) 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 1 -- 172.21.15.37:0/619972423 learned_addr learned my addr 172.21.15.37:0/619972423 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 10 monclient(hunting): get_auth_request con 0x7fb05809a540 auth_method 0 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb055d9b700 1 --2- 172.21.15.37:0/619972423 >> v2:172.21.15.143:3300/0 conn(0x7fb05809a0f0 0x7fb058137040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:41.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb055d9b700 10 monclient(hunting): get_auth_request con 0x7fb05809a0f0 auth_method 0 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb055d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb055d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb055d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 10 monclient(hunting): handle_auth_done global_id 34665 payload 274 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 1 -- 172.21.15.37:0/619972423 >> v2:172.21.15.143:3300/0 conn(0x7fb05809a0f0 msgr2=0x7fb058137040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 1 --2- 172.21.15.37:0/619972423 >> v2:172.21.15.143:3300/0 conn(0x7fb05809a0f0 0x7fb058137040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 1 -- 172.21.15.37:0/619972423 >> v1:172.21.15.143:6789/0 conn(0x7fb05809cca0 legacy=0x7fb0580999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 10 monclient: _finish_hunting 0 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 1 monclient: found mon.noname-a 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 1 -- 172.21.15.37:0/619972423 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05809bb40 con 0x7fb05809a540 2024-09-16T07:22:41.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb056d9d700 1 --2- 172.21.15.37:0/619972423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a540 0x7fb058139590 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb0580a37a0 tx=0x7fb05000ad90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a09f36c9ea926a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 1 -- 172.21.15.37:0/619972423 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05000cb70 con 0x7fb05809a540 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 10 monclient: dump: 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:41.510 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 10 monclient: _finish_auth 0 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 10 monclient: _check_auth_tickets 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 5 monclient: authenticate success, global_id 34665 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 1 -- 172.21.15.37:0/619972423 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05000ccd0 con 0x7fb05809a540 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 4 set_mon_vals no callback set 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 1 -- 172.21.15.37:0/619972423 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050018620 con 0x7fb05809a540 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05559a700 10 monclient: dump: 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:41.511 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 10 monclient: shutdown 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 -- 172.21.15.37:0/619972423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a540 msgr2=0x7fb058139590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 --2- 172.21.15.37:0/619972423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a540 0x7fb058139590 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb0580a37a0 tx=0x7fb05000ad90 comp rx=0 tx=0).stop 2024-09-16T07:22:41.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 -- 172.21.15.37:0/619972423 shutdown_connections 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 --2- 172.21.15.37:0/619972423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a540 0x7fb058139590 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 --2- 172.21.15.37:0/619972423 >> v2:172.21.15.143:3300/0 conn(0x7fb05809a0f0 0x7fb058137040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 -- 172.21.15.37:0/619972423 >> 172.21.15.37:0/619972423 conn(0x7fb0580a3c60 msgr2=0x7fb0580a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 -- 172.21.15.37:0/619972423 shutdown_connections 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.501+0000 7fb05759e700 1 -- 172.21.15.37:0/619972423 wait complete. 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient: build_initial_monmap 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient: monmap: 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:41.513 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:41.514 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:41.514 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:41.514 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:41.514 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:41.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 1 Processor -- start 2024-09-16T07:22:41.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 1 -- start start 2024-09-16T07:22:41.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient: init 2024-09-16T07:22:41.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:41.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:22:41.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a0f0 0x7fb0581c4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:41.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient(hunting): picked mon.noname-a con 0x7fb05809a0f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:41.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05809a540 0x7fb0581c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:41.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient(hunting): picked mon.noname-b con 0x7fb05809a540 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb05809cca0 0x7fb0581c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient(hunting): picked mon.noname-c con 0x7fb05809cca0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581a8d80 con 0x7fb05809a0f0 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb058070b30 con 0x7fb05809a540 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0580724c0 con 0x7fb05809cca0 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient(hunting): authenticate will time out at 1081.993688s 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a0f0 0x7fb0581c4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a0f0 0x7fb0581c4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51972/0 (socket says 172.21.15.37:51972) 2024-09-16T07:22:41.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 1 -- 172.21.15.37:0/356771045 learned_addr learned my addr 172.21.15.37:0/356771045 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 10 monclient(hunting): get_auth_request con 0x7fb05809a0f0 auth_method 0 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb056d9d700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb05809cca0 0x7fb0581c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb056d9d700 10 monclient(hunting): get_auth_request con 0x7fb05809cca0 auth_method 0 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb056d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb056d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb056d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:41.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb055d9b700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05809a540 0x7fb0581c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb055d9b700 10 monclient(hunting): get_auth_request con 0x7fb05809a540 auth_method 0 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb055d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb055d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb055d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb056d9d700 1 -- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb05809cca0 msgr2=0x7fb0581c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb056d9d700 1 -- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb05809cca0 msgr2=0x7fb0581c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb056d9d700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb05809cca0 0x7fb0581c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 10 monclient(hunting): handle_auth_done global_id 34668 payload 824 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb056d9d700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb05809cca0 0x7fb0581c7c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 1 -- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb05809cca0 msgr2=0x7fb0581c7c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb05809cca0 0x7fb0581c7c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 1 -- 172.21.15.37:0/356771045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05809a540 msgr2=0x7fb0581c1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:41.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05809a540 0x7fb0581c1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 10 monclient: _finish_hunting 0 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 1 monclient: found mon.noname-a 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 1 -- 172.21.15.37:0/356771045 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05000a710 con 0x7fb05809a0f0 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05659c700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a0f0 0x7fb0581c4730 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb04000c530 tx=0x7fb04000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 1 -- 172.21.15.37:0/356771045 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0400052d0 con 0x7fb05809a0f0 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 10 monclient: dump: 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:41.521 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 10 monclient: _finish_auth 0 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 10 monclient: _check_auth_tickets 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 5 monclient: authenticate success, global_id 34668 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 1 -- 172.21.15.37:0/356771045 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb040005430 con 0x7fb05809a0f0 2024-09-16T07:22:41.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 1 -- 172.21.15.37:0/356771045 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0400055a0 con 0x7fb05809a0f0 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient: _renew_subs 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 1 -- 172.21.15.37:0/356771045 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0581a6b40 con 0x7fb05809a0f0 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb054d99700 4 set_mon_vals no callback set 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb04effd700 10 monclient: dump: 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:41.523 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient: _renew_subs 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.505+0000 7fb05759e700 1 -- 172.21.15.37:0/356771045 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0581a9620 con 0x7fb05809a0f0 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.509+0000 7fb05759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.509+0000 7fb05759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.509+0000 7fb05759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.509+0000 7fb04effd700 1 -- 172.21.15.37:0/356771045 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fb0400038e0 con 0x7fb05809a0f0 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.509+0000 7fb05759e700 1 -- 172.21.15.37:0/356771045 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0581a8ef0 con 0x7fb05809a0f0 2024-09-16T07:22:41.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.509+0000 7fb04effd700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb03c035600 0x7fb03c037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:41.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.509+0000 7fb04effd700 1 -- 172.21.15.37:0/356771045 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fb040046fa0 con 0x7fb05809a0f0 2024-09-16T07:22:41.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.509+0000 7fb055d9b700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb03c035600 0x7fb03c037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:41.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.509+0000 7fb055d9b700 10 monclient: get_auth_request con 0x7fb03c035600 auth_method 0 2024-09-16T07:22:41.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.509+0000 7fb055d9b700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb03c035600 0x7fb03c037ac0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb0581a76d0 tx=0x7fb048003a40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:41.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.517+0000 7fb04effd700 1 -- 172.21.15.37:0/356771045 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb04000e280 con 0x7fb05809a0f0 2024-09-16T07:22:41.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.517+0000 7fb04effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:41.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.517+0000 7fb04effd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:41.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.757+0000 7fb05759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:22:41.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.757+0000 7fb05759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:41.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.757+0000 7fb05759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:41.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.757+0000 7fb05759e700 1 -- 172.21.15.37:0/356771045 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb0581a91d0 con 0x7fb05809a0f0 2024-09-16T07:22:41.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.761+0000 7fb04effd700 1 -- 172.21.15.37:0/356771045 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fb040017080 con 0x7fb05809a0f0 2024-09-16T07:22:41.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.761+0000 7fb04effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:41.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.761+0000 7fb04effd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:41.763 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":91,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.761+0000 7fb05759e700 1 -- 172.21.15.37:0/356771045 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb03c035600 msgr2=0x7fb03c037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.761+0000 7fb05759e700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb03c035600 0x7fb03c037ac0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb0581a76d0 tx=0x7fb048003a40 comp rx=0 tx=0).stop 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 10 monclient: shutdown 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 1 -- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a0f0 msgr2=0x7fb0581c4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a0f0 0x7fb0581c4730 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb04000c530 tx=0x7fb04000c8f0 comp rx=0 tx=0).stop 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 1 -- 172.21.15.37:0/356771045 shutdown_connections 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb03c035600 0x7fb03c037ac0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb05809cca0 0x7fb0581c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb05809a540 0x7fb0581c1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 1 --2- 172.21.15.37:0/356771045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05809a0f0 0x7fb0581c4730 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:41.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 1 -- 172.21.15.37:0/356771045 >> 172.21.15.37:0/356771045 conn(0x7fb0580a3c60 msgr2=0x7fb05806f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:41.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 1 -- 172.21.15.37:0/356771045 shutdown_connections 2024-09-16T07:22:41.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:41.765+0000 7fb05759e700 1 -- 172.21.15.37:0/356771045 wait complete. 2024-09-16T07:22:42.864 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:42.865 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27090]: starting. 2024-09-16T07:22:42.865 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27090]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27053 2024-09-16T07:22:42.865 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27090]: created object 0... 2024-09-16T07:22:42.865 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27090]: created object 25... 2024-09-16T07:22:42.865 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27090]: created object 49... 2024-09-16T07:22:42.865 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27090]: finishing. 2024-09-16T07:22:42.865 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27090]: shutting down. 2024-09-16T07:22:42.895 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:42.895 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27092]: starting. 2024-09-16T07:22:42.895 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27092]: listing objects. 2024-09-16T07:22:42.895 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27092]: listed object 0... 2024-09-16T07:22:42.895 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27092]: listed object 25... 2024-09-16T07:22:42.895 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27092]: saw 50 objects 2024-09-16T07:22:42.896 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27092]: shutting down. 2024-09-16T07:22:43.717 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:43.717 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27091]: starting. 2024-09-16T07:22:43.717 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27091]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27053 2024-09-16T07:22:43.717 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27091]: shutting down. 2024-09-16T07:22:43.719 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:43.719 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:43.719 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:22:43.722 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:22:44.782 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:22:44.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.893+0000 7feab65d5700 10 monclient: get_monmap_and_config 2024-09-16T07:22:44.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.893+0000 7feab65d5700 10 monclient: build_initial_monmap 2024-09-16T07:22:44.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.893+0000 7feab65d5700 10 monclient: monmap: 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:44.898033+0000 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:44.898033+0000 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:44.898 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:44.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.893+0000 7feab65d5700 1 Processor -- start 2024-09-16T07:22:44.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.893+0000 7feab65d5700 1 -- start start 2024-09-16T07:22:44.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.893+0000 7feab65d5700 10 monclient: init 2024-09-16T07:22:44.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.893+0000 7feab65d5700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:44.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.893+0000 7feab65d5700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-16T07:22:44.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feab00a0e90 addr v1:172.21.15.37:6789/0 2024-09-16T07:22:44.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a04d0 0x7feab00a1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:44.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 10 monclient(hunting): picked mon.noname-b con 0x7feab00a04d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7feab00a2d50 0x7feab00a31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 10 monclient(hunting): picked mon.noname-a con 0x7feab00a2d50 addr v2:172.21.15.37:3300/0 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab00a0900 con 0x7feab00a0e90 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab00a17b0 con 0x7feab00a2d50 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab006cf50 con 0x7feab00a04d0 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 10 monclient(hunting): authenticate will time out at 1085.384133s 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7feab00a2d50 0x7feab00a31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab4dd2700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a04d0 0x7feab00a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab55d3700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7feab00a0e90 0x7feab00a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:59338/0 (socket says 172.21.15.37:59338) 2024-09-16T07:22:44.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab55d3700 1 -- 172.21.15.37:0/16608642 learned_addr learned my addr 172.21.15.37:0/16608642 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 10 monclient(hunting): get_auth_request con 0x7feab00a2d50 auth_method 0 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab4dd2700 10 monclient(hunting): get_auth_request con 0x7feab00a04d0 auth_method 0 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab4dd2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab4dd2700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab4dd2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab4dd2700 1 -- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a04d0 msgr2=0x7feab00a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab4dd2700 1 -- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a04d0 msgr2=0x7feab00a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:22:44.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab4dd2700 1 --2- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a04d0 0x7feab00a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab4dd2700 1 --2- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a04d0 0x7feab00a1270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 1 -- 172.21.15.37:0/16608642 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711226226 0 0) 0x7feab00a0900 con 0x7feab00a0e90 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient(hunting): _negotiate 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient(hunting): my global_id is 34692 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 1 -- 172.21.15.37:0/16608642 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea9c0035d0 con 0x7feab00a0e90 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 10 monclient(hunting): handle_auth_done global_id 34689 payload 274 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 1 -- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a04d0 msgr2=0x7feab00a1270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 1 --2- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a04d0 0x7feab00a1270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 1 -- 172.21.15.37:0/16608642 >> v1:172.21.15.37:6789/0 conn(0x7feab00a0e90 legacy=0x7feab00a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 10 monclient: _finish_hunting 0 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 1 monclient: found mon.noname-a 2024-09-16T07:22:44.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 1 -- 172.21.15.37:0/16608642 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab00a36e0 con 0x7feab00a2d50 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab5dd4700 1 --2- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a2d50 0x7feab00a31a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009d90 tx=0x7feaa800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2bf50681178901 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 1 -- 172.21.15.37:0/16608642 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa800e070 con 0x7feab00a2d50 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient: dump: 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:44.906 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:44.907 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:44.907 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:44.907 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient: _finish_auth 0 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient: _check_auth_tickets 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 1 -- 172.21.15.37:0/16608642 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa800bb20 con 0x7feab00a2d50 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 5 monclient: authenticate success, global_id 34689 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:44.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 4 set_mon_vals no callback set 2024-09-16T07:22:44.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 1 -- 172.21.15.37:0/16608642 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8012430 con 0x7feab00a2d50 2024-09-16T07:22:44.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:44.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:44.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feaa3fff700 10 monclient: dump: 2024-09-16T07:22:44.909 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:44.909 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:44.909 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:44.909 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:44.909 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 10 monclient: shutdown 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 -- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a2d50 msgr2=0x7feab00a31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 --2- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a2d50 0x7feab00a31a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009d90 tx=0x7feaa800b330 comp rx=0 tx=0).stop 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 -- 172.21.15.37:0/16608642 shutdown_connections 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 --2- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a2d50 0x7feab00a31a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 --2- 172.21.15.37:0/16608642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a04d0 0x7feab00a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 -- 172.21.15.37:0/16608642 >> 172.21.15.37:0/16608642 conn(0x7feab00a4740 msgr2=0x7feab00a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:44.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 -- 172.21.15.37:0/16608642 shutdown_connections 2024-09-16T07:22:44.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.897+0000 7feab65d5700 1 -- 172.21.15.37:0/16608642 wait complete. 2024-09-16T07:22:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient: build_initial_monmap 2024-09-16T07:22:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient: monmap: 2024-09-16T07:22:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 1 Processor -- start 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 1 -- start start 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient: init 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a04d0 0x7feab013bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient(hunting): picked mon.noname-a con 0x7feab00a04d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:44.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0e90 0x7feab013c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient(hunting): picked mon.noname-c con 0x7feab00a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feab00a2d50 0x7feab0122c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient(hunting): picked mon.noname-b con 0x7feab00a2d50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab006d780 con 0x7feab00a04d0 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab013c9e0 con 0x7feab00a2d50 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab0126970 con 0x7feab00a0e90 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient(hunting): authenticate will time out at 1085.390142s 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a04d0 0x7feab013bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab4dd2700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0e90 0x7feab013c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a04d0 0x7feab013bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52026/0 (socket says 172.21.15.37:52026) 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab4dd2700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0e90 0x7feab013c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:36180/0 (socket says 172.21.15.37:36180) 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 1 -- 172.21.15.37:0/1098664138 learned_addr learned my addr 172.21.15.37:0/1098664138 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 10 monclient(hunting): get_auth_request con 0x7feab00a04d0 auth_method 0 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab4dd2700 10 monclient(hunting): get_auth_request con 0x7feab00a0e90 auth_method 0 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab4dd2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab4dd2700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab4dd2700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab5dd4700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feab00a2d50 0x7feab0122c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab5dd4700 10 monclient(hunting): get_auth_request con 0x7feab00a2d50 auth_method 0 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab5dd4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab5dd4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab5dd4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab4dd2700 1 -- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0e90 msgr2=0x7feab013c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab4dd2700 1 -- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0e90 msgr2=0x7feab013c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab4dd2700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0e90 0x7feab013c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab5dd4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab5dd4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab4dd2700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0e90 0x7feab013c0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab5dd4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 10 monclient(hunting): handle_auth_done global_id 34695 payload 824 2024-09-16T07:22:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 1 -- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0e90 msgr2=0x7feab013c0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0e90 0x7feab013c0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 1 -- 172.21.15.37:0/1098664138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feab00a2d50 msgr2=0x7feab0122c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feab00a2d50 0x7feab0122c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 10 monclient: _finish_hunting 0 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 1 monclient: found mon.noname-a 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 1 -- 172.21.15.37:0/1098664138 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa8009a70 con 0x7feab00a04d0 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab55d3700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a04d0 0x7feab013bb80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feaa400c700 tx=0x7feaa400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 1 -- 172.21.15.37:0/1098664138 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa4005d90 con 0x7feab00a04d0 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 10 monclient: dump: 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 10 monclient: _finish_auth 0 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 5 monclient: authenticate success, global_id 34695 2024-09-16T07:22:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient: _renew_subs 2024-09-16T07:22:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 1 -- 172.21.15.37:0/1098664138 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa40070d0 con 0x7feab00a04d0 2024-09-16T07:22:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 1 -- 172.21.15.37:0/1098664138 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab0125440 con 0x7feab00a04d0 2024-09-16T07:22:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 1 -- 172.21.15.37:0/1098664138 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa4007480 con 0x7feab00a04d0 2024-09-16T07:22:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa1ffb700 10 monclient: dump: 2024-09-16T07:22:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feaa37fe700 4 set_mon_vals no callback set 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient: _renew_subs 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.901+0000 7feab65d5700 1 -- 172.21.15.37:0/1098664138 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab0124870 con 0x7feab00a04d0 2024-09-16T07:22:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.905+0000 7feaa1ffb700 1 -- 172.21.15.37:0/1098664138 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7feaa40075e0 con 0x7feab00a04d0 2024-09-16T07:22:44.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.905+0000 7feaa1ffb700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fea94035630 0x7fea94037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:44.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.905+0000 7feab65d5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:44.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.905+0000 7feab65d5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:44.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.905+0000 7feab65d5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:44.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.905+0000 7feab65d5700 1 -- 172.21.15.37:0/1098664138 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab00a1270 con 0x7feab00a04d0 2024-09-16T07:22:44.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.905+0000 7feaa1ffb700 1 -- 172.21.15.37:0/1098664138 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 4738+0+0 (secure 0 0 0) 0x7feaa40471b0 con 0x7feab00a04d0 2024-09-16T07:22:44.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.905+0000 7feab4dd2700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fea94035630 0x7fea94037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:44.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.905+0000 7feab4dd2700 10 monclient: get_auth_request con 0x7fea94035630 auth_method 0 2024-09-16T07:22:44.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.905+0000 7feab4dd2700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fea94035630 0x7fea94037af0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feab0124470 tx=0x7feaac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:44.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.913+0000 7feaa1ffb700 1 -- 172.21.15.37:0/1098664138 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7feaa4017020 con 0x7feab00a04d0 2024-09-16T07:22:44.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.913+0000 7feaa1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:44.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:44.913+0000 7feaa1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:45.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.145+0000 7feab65d5700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:22:45.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.145+0000 7feab65d5700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:45.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.145+0000 7feab65d5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:45.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.145+0000 7feab65d5700 1 -- 172.21.15.37:0/1098664138 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feab00a2110 con 0x7feab00a04d0 2024-09-16T07:22:45.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feaa1ffb700 1 -- 172.21.15.37:0/1098664138 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7feaa400e280 con 0x7feab00a04d0 2024-09-16T07:22:45.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feaa1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:45.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feaa1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:45.151 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":94,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:22:45.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 -- 172.21.15.37:0/1098664138 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fea94035630 msgr2=0x7fea94037af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:45.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fea94035630 0x7fea94037af0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feab0124470 tx=0x7feaac008040 comp rx=0 tx=0).stop 2024-09-16T07:22:45.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 10 monclient: shutdown 2024-09-16T07:22:45.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:45.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 -- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a04d0 msgr2=0x7feab013bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:45.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a04d0 0x7feab013bb80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feaa400c700 tx=0x7feaa400ca10 comp rx=0 tx=0).stop 2024-09-16T07:22:45.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 -- 172.21.15.37:0/1098664138 shutdown_connections 2024-09-16T07:22:45.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feab00a2d50 0x7feab0122c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:45.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fea94035630 0x7fea94037af0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:45.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0e90 0x7feab013c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:45.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 --2- 172.21.15.37:0/1098664138 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab00a04d0 0x7feab013bb80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:45.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 -- 172.21.15.37:0/1098664138 >> 172.21.15.37:0/1098664138 conn(0x7feab00a4740 msgr2=0x7feab009e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:45.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 -- 172.21.15.37:0/1098664138 shutdown_connections 2024-09-16T07:22:45.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:45.149+0000 7feab65d5700 1 -- 172.21.15.37:0/1098664138 wait complete. 2024-09-16T07:22:45.788 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27145]: starting. 2024-09-16T07:22:45.788 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27145]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27144 2024-09-16T07:22:45.788 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27145]: created object 0... 2024-09-16T07:22:45.788 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27145]: created object 25... 2024-09-16T07:22:45.788 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27145]: created object 49... 2024-09-16T07:22:45.788 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27145]: finishing. 2024-09-16T07:22:45.788 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27145]: shutting down. 2024-09-16T07:22:46.660 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27146]: starting. 2024-09-16T07:22:46.660 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27146]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27144 2024-09-16T07:22:46.660 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27146]: shutting down. 2024-09-16T07:22:48.167 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:22:48.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient: get_monmap_and_config 2024-09-16T07:22:48.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient: build_initial_monmap 2024-09-16T07:22:48.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient: monmap: 2024-09-16T07:22:48.287 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:48.287 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:48.287 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:48.287506+0000 2024-09-16T07:22:48.287 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:48.287506+0000 2024-09-16T07:22:48.288 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:48.288 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:48.288 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:48.288 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:48.288 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:48.288 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:48.288 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:48.288 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:48.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 1 Processor -- start 2024-09-16T07:22:48.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 1 -- start start 2024-09-16T07:22:48.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient: init 2024-09-16T07:22:48.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:48.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T07:22:48.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86380a0e90 addr v1:172.21.15.143:6789/0 2024-09-16T07:22:48.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86380a04d0 0x7f86380a1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:48.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient(hunting): picked mon.noname-c con 0x7f86380a04d0 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a2d50 0x7f86380a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient(hunting): picked mon.noname-b con 0x7f86380a2d50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86380a0900 con 0x7f86380a0e90 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86380a17b0 con 0x7f86380a04d0 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863806cf50 con 0x7f86380a2d50 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient(hunting): authenticate will time out at 1088.773573s 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a2d50 0x7f86380a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86380a04d0 0x7f86380a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a2d50 0x7f86380a3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:36184/0 (socket says 172.21.15.37:36184) 2024-09-16T07:22:48.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 1 -- 172.21.15.37:0/4015328995 learned_addr learned my addr 172.21.15.37:0/4015328995 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 10 monclient(hunting): get_auth_request con 0x7f86380a2d50 auth_method 0 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 10 monclient(hunting): get_auth_request con 0x7f86380a04d0 auth_method 0 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 1 -- 172.21.15.37:0/4015328995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a2d50 msgr2=0x7f86380a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 1 -- 172.21.15.37:0/4015328995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a2d50 msgr2=0x7f86380a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 1 --2- 172.21.15.37:0/4015328995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a2d50 0x7f86380a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86377fe700 1 --2- 172.21.15.37:0/4015328995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a2d50 0x7f86380a3180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:48.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 10 monclient(hunting): handle_auth_done global_id 5611 payload 274 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 1 -- 172.21.15.37:0/4015328995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a2d50 msgr2=0x7f86380a3180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 1 --2- 172.21.15.37:0/4015328995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a2d50 0x7f86380a3180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 1 -- 172.21.15.37:0/4015328995 >> v1:172.21.15.143:6789/0 conn(0x7f86380a0e90 legacy=0x7f86380a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 10 monclient: _finish_hunting 0 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 1 monclient: found mon.noname-c 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:22:48.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 1 -- 172.21.15.37:0/4015328995 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86380a36c0 con 0x7f86380a04d0 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f86367fc700 1 --2- 172.21.15.37:0/4015328995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86380a04d0 0x7f86380a1270 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8620009880 tx=0x7f8620009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b042544e64534c2e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 1 -- 172.21.15.37:0/4015328995 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620011830 con 0x7f86380a04d0 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 10 monclient: dump: 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:48.295 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 10 monclient: _finish_auth 0 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 5 monclient: authenticate success, global_id 5611 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 1 -- 172.21.15.37:0/4015328995 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8620011990 con 0x7f86380a04d0 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 1 -- 172.21.15.37:0/4015328995 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620011d20 con 0x7f86380a04d0 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:48.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8635ffb700 10 monclient: dump: 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:48.297 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:48.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:48.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.285+0000 7f8637fff700 4 set_mon_vals no callback set 2024-09-16T07:22:48.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient: shutdown 2024-09-16T07:22:48.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:48.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 -- 172.21.15.37:0/4015328995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86380a04d0 msgr2=0x7f86380a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:48.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 --2- 172.21.15.37:0/4015328995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86380a04d0 0x7f86380a1270 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8620009880 tx=0x7f8620009b90 comp rx=0 tx=0).stop 2024-09-16T07:22:48.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 -- 172.21.15.37:0/4015328995 shutdown_connections 2024-09-16T07:22:48.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 --2- 172.21.15.37:0/4015328995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a2d50 0x7f86380a3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:48.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 --2- 172.21.15.37:0/4015328995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86380a04d0 0x7f86380a1270 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:48.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 -- 172.21.15.37:0/4015328995 >> 172.21.15.37:0/4015328995 conn(0x7f86380a4740 msgr2=0x7f86380a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:48.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 -- 172.21.15.37:0/4015328995 shutdown_connections 2024-09-16T07:22:48.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 -- 172.21.15.37:0/4015328995 wait complete. 2024-09-16T07:22:48.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient: build_initial_monmap 2024-09-16T07:22:48.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient: monmap: 2024-09-16T07:22:48.303 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:48.303 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:48.303 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 Processor -- start 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 -- start start 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient: init 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:48.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a04d0 0x7f86381c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient(hunting): picked mon.noname-c con 0x7f86380a04d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86380a0e90 0x7f86381c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient(hunting): picked mon.noname-a con 0x7f86380a0e90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86380a2d50 0x7f86381c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient(hunting): picked mon.noname-b con 0x7f86380a2d50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86381a8ef0 con 0x7f86380a0e90 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86381ab330 con 0x7f86380a2d50 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863806c0b0 con 0x7f86380a04d0 2024-09-16T07:22:48.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8637fff700 10 monclient(hunting): authenticate will time out at 1088.778984s 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86380a0e90 0x7f86381c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8636ffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a04d0 0x7f86381c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86380a0e90 0x7f86381c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52034/0 (socket says 172.21.15.37:52034) 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 1 -- 172.21.15.37:0/4120527414 learned_addr learned my addr 172.21.15.37:0/4120527414 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 10 monclient(hunting): get_auth_request con 0x7f86380a0e90 auth_method 0 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8636ffd700 10 monclient(hunting): get_auth_request con 0x7f86380a04d0 auth_method 0 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8636ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8636ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8636ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:48.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:48.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:48.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:48.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8636ffd700 1 -- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a04d0 msgr2=0x7f86381c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T07:22:48.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8636ffd700 1 -- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a04d0 msgr2=0x7f86381c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:22:48.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8636ffd700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a04d0 0x7f86381c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:48.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f8636ffd700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a04d0 0x7f86381c9ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:48.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 10 monclient(hunting): handle_auth_done global_id 34716 payload 824 2024-09-16T07:22:48.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 1 -- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a04d0 msgr2=0x7f86381c9ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:22:48.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a04d0 0x7f86381c9ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 1 -- 172.21.15.37:0/4120527414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86380a2d50 msgr2=0x7f86381c6970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86380a2d50 0x7f86381c6970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 10 monclient: _finish_hunting 0 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 1 monclient: found mon.noname-a 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.289+0000 7f86367fc700 1 -- 172.21.15.37:0/4120527414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86200094b0 con 0x7f86380a0e90 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f86367fc700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86380a0e90 0x7f86381c5fb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8620009bd0 tx=0x7f8620004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 1 -- 172.21.15.37:0/4120527414 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86200038c0 con 0x7f86380a0e90 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 10 monclient: dump: 2024-09-16T07:22:48.308 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 10 monclient: _finish_auth 0 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:48.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 5 monclient: authenticate success, global_id 34716 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 10 monclient: _renew_subs 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 1 -- 172.21.15.37:0/4120527414 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8620003a40 con 0x7f86380a0e90 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 1 -- 172.21.15.37:0/4120527414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86381a9560 con 0x7f86380a0e90 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 1 -- 172.21.15.37:0/4120527414 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620003df0 con 0x7f86380a0e90 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 10 monclient: dump: 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:48.310 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f86357fa700 4 set_mon_vals no callback set 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 10 monclient: _renew_subs 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 1 -- 172.21.15.37:0/4120527414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86381aa350 con 0x7f86380a0e90 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:48.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8637fff700 1 -- 172.21.15.37:0/4120527414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86380a1270 con 0x7f86380a0e90 2024-09-16T07:22:48.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 1 -- 172.21.15.37:0/4120527414 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f8620030480 con 0x7f86380a0e90 2024-09-16T07:22:48.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f861c035600 0x7f861c037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:48.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f862f7fe700 1 -- 172.21.15.37:0/4120527414 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f8620021030 con 0x7f86380a0e90 2024-09-16T07:22:48.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8636ffd700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f861c035600 0x7f861c037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:48.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.293+0000 7f8636ffd700 10 monclient: get_auth_request con 0x7f861c035600 auth_method 0 2024-09-16T07:22:48.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.297+0000 7f8636ffd700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f861c035600 0x7f861c037ac0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8628001930 tx=0x7f8628009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:48.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.301+0000 7f862f7fe700 1 -- 172.21.15.37:0/4120527414 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8620019d80 con 0x7f86380a0e90 2024-09-16T07:22:48.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.301+0000 7f862f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:48.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.301+0000 7f862f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:48.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.533+0000 7f8637fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:22:48.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.533+0000 7f8637fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:48.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.533+0000 7f8637fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:48.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.533+0000 7f8637fff700 1 -- 172.21.15.37:0/4120527414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f86380a2110 con 0x7f86380a0e90 2024-09-16T07:22:48.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.533+0000 7f862f7fe700 1 -- 172.21.15.37:0/4120527414 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f862002b070 con 0x7f86380a0e90 2024-09-16T07:22:48.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.533+0000 7f862f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:48.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.533+0000 7f862f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:48.539 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":97,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:22:48.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 -- 172.21.15.37:0/4120527414 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f861c035600 msgr2=0x7f861c037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f861c035600 0x7f861c037ac0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8628001930 tx=0x7f8628009040 comp rx=0 tx=0).stop 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 10 monclient: shutdown 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 -- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86380a0e90 msgr2=0x7f86381c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86380a0e90 0x7f86381c5fb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8620009bd0 tx=0x7f8620004770 comp rx=0 tx=0).stop 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 -- 172.21.15.37:0/4120527414 shutdown_connections 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86380a2d50 0x7f86381c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86380a0e90 0x7f86381c5fb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f861c035600 0x7f861c037ac0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 --2- 172.21.15.37:0/4120527414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86380a04d0 0x7f86381c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 -- 172.21.15.37:0/4120527414 >> 172.21.15.37:0/4120527414 conn(0x7f86380a4740 msgr2=0x7f863809c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 -- 172.21.15.37:0/4120527414 shutdown_connections 2024-09-16T07:22:48.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:48.537+0000 7f8637fff700 1 -- 172.21.15.37:0/4120527414 wait complete. 2024-09-16T07:22:48.827 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:48.828 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27181]: starting. 2024-09-16T07:22:48.828 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27181]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27144 2024-09-16T07:22:48.828 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27181]: created object 0... 2024-09-16T07:22:48.828 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27181]: created object 25... 2024-09-16T07:22:48.828 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27181]: created object 49... 2024-09-16T07:22:48.828 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27181]: finishing. 2024-09-16T07:22:48.828 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27181]: shutting down. 2024-09-16T07:22:48.859 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:48.859 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27183]: starting. 2024-09-16T07:22:48.859 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27183]: listing objects. 2024-09-16T07:22:48.859 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27183]: listed object 0... 2024-09-16T07:22:48.859 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27183]: listed object 25... 2024-09-16T07:22:48.859 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27183]: saw 50 objects 2024-09-16T07:22:48.859 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27183]: shutting down. 2024-09-16T07:22:49.685 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:49.685 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27182]: starting. 2024-09-16T07:22:49.685 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27182]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27144 2024-09-16T07:22:49.685 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27182]: shutting down. 2024-09-16T07:22:49.686 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:49.686 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:49.686 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:22:49.688 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:22:51.554 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:22:51.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient: get_monmap_and_config 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient: build_initial_monmap 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient: monmap: 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:51.673385+0000 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:51.673385+0000 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:51.673 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:51.674 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:51.674 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:51.674 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:51.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 1 Processor -- start 2024-09-16T07:22:51.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 1 -- start start 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient: init 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb8071880 0x7f7bb80a2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient(hunting): picked mon.noname-b con 0x7f7bb8071880 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7bb80a31e0 0x7f7bb80a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient(hunting): picked mon.noname-c con 0x7f7bb80a31e0 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7bb80999e0 addr v1:172.21.15.143:6789/0 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb809c3e0 con 0x7f7bb80999e0 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb809d0e0 con 0x7f7bb80a31e0 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:51.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb809bb40 con 0x7f7bb8071880 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbe6a6700 10 monclient(hunting): authenticate will time out at 1092.159255s 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbd6a4700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb8071880 0x7f7bb80a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbd6a4700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb8071880 0x7f7bb80a2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50466/0 (socket says 172.21.15.37:50466) 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbd6a4700 1 -- 172.21.15.37:0/621131022 learned_addr learned my addr 172.21.15.37:0/621131022 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbd6a4700 10 monclient(hunting): get_auth_request con 0x7f7bb8071880 auth_method 0 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbd6a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbd6a4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbd6a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbcea3700 1 --2- 172.21.15.37:0/621131022 >> v2:172.21.15.143:3300/0 conn(0x7f7bb80a31e0 0x7f7bb80a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbcea3700 10 monclient(hunting): get_auth_request con 0x7f7bb80a31e0 auth_method 0 2024-09-16T07:22:51.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbcea3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbcea3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.669+0000 7f7bbcea3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbd6a4700 1 -- 172.21.15.37:0/621131022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb8071880 msgr2=0x7f7bb80a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbd6a4700 1 -- 172.21.15.37:0/621131022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb8071880 msgr2=0x7f7bb80a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbd6a4700 1 --2- 172.21.15.37:0/621131022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb8071880 0x7f7bb80a2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbd6a4700 1 --2- 172.21.15.37:0/621131022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb8071880 0x7f7bb80a2750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 10 monclient(hunting): handle_auth_done global_id 5632 payload 274 2024-09-16T07:22:51.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 1 -- 172.21.15.37:0/621131022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb8071880 msgr2=0x7f7bb80a2750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 1 --2- 172.21.15.37:0/621131022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb8071880 0x7f7bb80a2750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 1 -- 172.21.15.37:0/621131022 >> v1:172.21.15.143:6789/0 conn(0x7f7bb80999e0 legacy=0x7f7bb8099e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 10 monclient: _finish_hunting 0 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 1 monclient: found mon.noname-c 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 1 -- 172.21.15.37:0/621131022 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb809ab10 con 0x7f7bb80a31e0 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbcea3700 1 --2- 172.21.15.37:0/621131022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb80a31e0 0x7f7bb80a0a50 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009850 tx=0x7f7ba8009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b32be1953f25d6c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 1 -- 172.21.15.37:0/621131022 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8011830 con 0x7f7bb80a31e0 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 10 monclient: dump: 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:51.682 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 10 monclient: _finish_auth 0 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 10 monclient: _check_auth_tickets 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 5 monclient: authenticate success, global_id 5632 2024-09-16T07:22:51.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:51.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 1 -- 172.21.15.37:0/621131022 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba8011990 con 0x7f7bb80a31e0 2024-09-16T07:22:51.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:51.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 1 -- 172.21.15.37:0/621131022 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8011d20 con 0x7f7bb80a31e0 2024-09-16T07:22:51.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:51.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 4 set_mon_vals no callback set 2024-09-16T07:22:51.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:51.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:51.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bb7fff700 10 monclient: dump: 2024-09-16T07:22:51.684 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:51.684 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 10 monclient: shutdown 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/621131022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb80a31e0 msgr2=0x7f7bb80a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 1 --2- 172.21.15.37:0/621131022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb80a31e0 0x7f7bb80a0a50 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009850 tx=0x7f7ba8009b60 comp rx=0 tx=0).stop 2024-09-16T07:22:51.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/621131022 shutdown_connections 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 1 --2- 172.21.15.37:0/621131022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb80a31e0 0x7f7bb80a0a50 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 1 --2- 172.21.15.37:0/621131022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb8071880 0x7f7bb80a2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/621131022 >> 172.21.15.37:0/621131022 conn(0x7f7bb80a3ea0 msgr2=0x7f7bb80a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/621131022 shutdown_connections 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/621131022 wait complete. 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 10 monclient: build_initial_monmap 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.673+0000 7f7bbe6a6700 10 monclient: monmap: 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:51.686 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:51.687 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:51.687 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:51.687 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:51.687 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:51.687 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:51.688 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:51.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 1 Processor -- start 2024-09-16T07:22:51.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 1 -- start start 2024-09-16T07:22:51.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient: init 2024-09-16T07:22:51.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:51.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:22:51.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bb8071880 0x7f7bb81c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:51.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient(hunting): picked mon.noname-a con 0x7f7bb8071880 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:51.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb80999e0 0x7f7bb81c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:51.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient(hunting): picked mon.noname-c con 0x7f7bb80999e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:51.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb80a31e0 0x7f7bb811e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:51.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient(hunting): picked mon.noname-b con 0x7f7bb80a31e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:51.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:51.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb8121040 con 0x7f7bb8071880 2024-09-16T07:22:51.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:51.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb80724c0 con 0x7f7bb80a31e0 2024-09-16T07:22:51.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:51.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb80a08d0 con 0x7f7bb80999e0 2024-09-16T07:22:51.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:51.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient(hunting): authenticate will time out at 1092.166005s 2024-09-16T07:22:51.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb80999e0 0x7f7bb81c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:51.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bb8071880 0x7f7bb81c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:51.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb80999e0 0x7f7bb81c88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50480/0 (socket says 172.21.15.37:50480) 2024-09-16T07:22:51.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 1 -- 172.21.15.37:0/976702005 learned_addr learned my addr 172.21.15.37:0/976702005 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:51.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 10 monclient(hunting): get_auth_request con 0x7f7bb80999e0 auth_method 0 2024-09-16T07:22:51.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:51.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:51.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:51.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 10 monclient(hunting): get_auth_request con 0x7f7bb8071880 auth_method 0 2024-09-16T07:22:51.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:51.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:51.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:51.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbdea5700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb80a31e0 0x7f7bb811e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:51.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:51.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:51.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 1 -- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb80999e0 msgr2=0x7f7bb81c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T07:22:51.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 1 -- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb80999e0 msgr2=0x7f7bb81c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:22:51.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb80999e0 0x7f7bb81c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:51.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:51.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbdea5700 10 monclient(hunting): get_auth_request con 0x7f7bb80a31e0 auth_method 0 2024-09-16T07:22:51.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbdea5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:51.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbdea5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:51.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbdea5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:51.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbcea3700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb80999e0 0x7f7bb81c88a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:51.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbdea5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:51.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbdea5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:51.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbdea5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:51.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 10 monclient(hunting): handle_auth_done global_id 34728 payload 824 2024-09-16T07:22:51.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 1 -- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb80999e0 msgr2=0x7f7bb81c88a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:22:51.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb80999e0 0x7f7bb81c88a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:51.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 1 -- 172.21.15.37:0/976702005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb80a31e0 msgr2=0x7f7bb811e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:51.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb80a31e0 0x7f7bb811e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:51.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 10 monclient: _finish_hunting 0 2024-09-16T07:22:51.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 1 monclient: found mon.noname-a 2024-09-16T07:22:51.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:51.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:51.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 1 -- 172.21.15.37:0/976702005 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba80094b0 con 0x7f7bb8071880 2024-09-16T07:22:51.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbd6a4700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bb8071880 0x7f7bb81c8360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00e810 tx=0x7f7bac00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:51.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 1 -- 172.21.15.37:0/976702005 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac004220 con 0x7f7bb8071880 2024-09-16T07:22:51.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:51.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:51.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 10 monclient: dump: 2024-09-16T07:22:51.707 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:51.708 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:51.708 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:51.708 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:51.708 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:51.708 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:51.708 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:51.709 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:51.709 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:51.709 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:51.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 10 monclient: _finish_auth 0 2024-09-16T07:22:51.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:22:51.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:51.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 5 monclient: authenticate success, global_id 34728 2024-09-16T07:22:51.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 1 -- 172.21.15.37:0/976702005 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bac004380 con 0x7f7bb8071880 2024-09-16T07:22:51.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient: _renew_subs 2024-09-16T07:22:51.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:51.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/976702005 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb8070890 con 0x7f7bb8071880 2024-09-16T07:22:51.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:51.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 1 -- 172.21.15.37:0/976702005 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0036c0 con 0x7f7bb8071880 2024-09-16T07:22:51.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:51.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:51.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb77fe700 4 set_mon_vals no callback set 2024-09-16T07:22:51.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.677+0000 7f7bb5ffb700 10 monclient: dump: 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bbe6a6700 10 monclient: _renew_subs 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bbe6a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/976702005 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb8122760 con 0x7f7bb8071880 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bb5ffb700 1 -- 172.21.15.37:0/976702005 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f7bac003820 con 0x7f7bb8071880 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bb5ffb700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7ba4035600 0x7f7ba4037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bb5ffb700 1 -- 172.21.15.37:0/976702005 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f7bac047480 con 0x7f7bb8071880 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bbcea3700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7ba4035600 0x7f7ba4037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bbe6a6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bbe6a6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bbe6a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/976702005 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb8120db0 con 0x7f7bb8071880 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bbcea3700 10 monclient: get_auth_request con 0x7f7ba4035600 auth_method 0 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.681+0000 7f7bbcea3700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7ba4035600 0x7f7ba4037ac0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8120000 tx=0x7f7ba8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.693+0000 7f7bb5ffb700 1 -- 172.21.15.37:0/976702005 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7bac045e60 con 0x7f7bb8071880 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.693+0000 7f7bb5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.693+0000 7f7bb5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:51.844 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27236]: starting. 2024-09-16T07:22:51.844 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27236]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27235 2024-09-16T07:22:51.844 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27236]: created object 0... 2024-09-16T07:22:51.844 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27236]: created object 25... 2024-09-16T07:22:51.844 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27236]: created object 49... 2024-09-16T07:22:51.844 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27236]: finishing. 2024-09-16T07:22:51.844 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27236]: shutting down. 2024-09-16T07:22:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.945+0000 7f7bbe6a6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:22:51.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.945+0000 7f7bbe6a6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:51.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.945+0000 7f7bbe6a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:51.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.945+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/976702005 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7bb80a2750 con 0x7f7bb8071880 2024-09-16T07:22:51.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.945+0000 7f7bb5ffb700 1 -- 172.21.15.37:0/976702005 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f7bac029620 con 0x7f7bb8071880 2024-09-16T07:22:51.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.945+0000 7f7bb5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:51.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.945+0000 7f7bb5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:51.949 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":101,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:22:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/976702005 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7ba4035600 msgr2=0x7f7ba4037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7ba4035600 0x7f7ba4037ac0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8120000 tx=0x7f7ba8008030 comp rx=0 tx=0).stop 2024-09-16T07:22:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 10 monclient: shutdown 2024-09-16T07:22:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bb8071880 msgr2=0x7f7bb81c8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bb8071880 0x7f7bb81c8360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00e810 tx=0x7f7bac00eb20 comp rx=0 tx=0).stop 2024-09-16T07:22:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/976702005 shutdown_connections 2024-09-16T07:22:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bb80a31e0 0x7f7bb811e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7ba4035600 0x7f7ba4037ac0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bb80999e0 0x7f7bb81c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 --2- 172.21.15.37:0/976702005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bb8071880 0x7f7bb81c8360 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/976702005 >> 172.21.15.37:0/976702005 conn(0x7f7bb80a3ea0 msgr2=0x7f7bb8070dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/976702005 shutdown_connections 2024-09-16T07:22:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:51.949+0000 7f7bbe6a6700 1 -- 172.21.15.37:0/976702005 wait complete. 2024-09-16T07:22:52.712 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27237]: starting. 2024-09-16T07:22:52.712 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27237]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27235 2024-09-16T07:22:52.712 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27237]: shutting down. 2024-09-16T07:22:54.884 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:54.885 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27272]: starting. 2024-09-16T07:22:54.885 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27272]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27235 2024-09-16T07:22:54.885 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27272]: created object 0... 2024-09-16T07:22:54.885 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27272]: created object 25... 2024-09-16T07:22:54.885 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27272]: created object 49... 2024-09-16T07:22:54.885 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27272]: finishing. 2024-09-16T07:22:54.885 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27272]: shutting down. 2024-09-16T07:22:54.913 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:54.913 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27274]: starting. 2024-09-16T07:22:54.913 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27274]: listing objects. 2024-09-16T07:22:54.913 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27274]: listed object 0... 2024-09-16T07:22:54.913 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27274]: listed object 25... 2024-09-16T07:22:54.913 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27274]: saw 50 objects 2024-09-16T07:22:54.913 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27274]: shutting down. 2024-09-16T07:22:54.966 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:22:55.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient: get_monmap_and_config 2024-09-16T07:22:55.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient: build_initial_monmap 2024-09-16T07:22:55.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient: monmap: 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:55.084729+0000 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:55.084729+0000 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 1 Processor -- start 2024-09-16T07:22:55.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 1 -- start start 2024-09-16T07:22:55.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient: init 2024-09-16T07:22:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-16T07:22:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f81fc09ce00 0x7f81fc0999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient(hunting): picked mon.noname-a con 0x7f81fc09ce00 addr v2:172.21.15.37:3300/0 2024-09-16T07:22:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81fc099f20 addr v1:172.21.15.37:6789/0 2024-09-16T07:22:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc1398d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient(hunting): picked mon.noname-b con 0x7f81fc09a330 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc09ab10 con 0x7f81fc099f20 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81fc09c060 con 0x7f81fc09ce00 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc09ace0 con 0x7f81fc09a330 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8203ecc700 10 monclient(hunting): authenticate will time out at 1095.570938s 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc1398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f81fc09ce00 0x7f81fc0999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc1398d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50508/0 (socket says 172.21.15.37:50508) 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 1 -- 172.21.15.37:0/1667211911 learned_addr learned my addr 172.21.15.37:0/1667211911 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 10 monclient(hunting): get_auth_request con 0x7f81fc09a330 auth_method 0 2024-09-16T07:22:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 10 monclient(hunting): get_auth_request con 0x7f81fc09ce00 auth_method 0 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 1 -- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 msgr2=0x7f81fc1398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 1 -- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 msgr2=0x7f81fc1398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 1 --2- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc1398d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8202469700 1 --2- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc1398d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 10 monclient(hunting): handle_auth_done global_id 34752 payload 274 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 1 -- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 msgr2=0x7f81fc1398d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 1 --2- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc1398d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 1 -- 172.21.15.37:0/1667211911 >> v1:172.21.15.37:6789/0 conn(0x7f81fc099f20 legacy=0x7f81fc1361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 10 monclient: _finish_hunting 0 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 1 monclient: found mon.noname-a 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.081+0000 7f8201c68700 1 -- 172.21.15.37:0/1667211911 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc0a0250 con 0x7f81fc09ce00 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201c68700 1 --2- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 0x7f81fc0999e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009850 tx=0x7f81f8009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5785cae6c3823c26 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 1 -- 172.21.15.37:0/1667211911 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f8011830 con 0x7f81fc09ce00 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 10 monclient: dump: 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 10 monclient: _finish_auth 0 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 10 monclient: _check_auth_tickets 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 5 monclient: authenticate success, global_id 34752 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 1 -- 172.21.15.37:0/1667211911 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f8011990 con 0x7f81fc09ce00 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 4 set_mon_vals no callback set 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 1 -- 172.21.15.37:0/1667211911 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f8011d20 con 0x7f81fc09ce00 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8200c66700 10 monclient: dump: 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient: shutdown 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 -- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 msgr2=0x7f81fc0999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 --2- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 0x7f81fc0999e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009850 tx=0x7f81f8009b60 comp rx=0 tx=0).stop 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 -- 172.21.15.37:0/1667211911 shutdown_connections 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 --2- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc1398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 --2- 172.21.15.37:0/1667211911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 0x7f81fc0999e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 -- 172.21.15.37:0/1667211911 >> 172.21.15.37:0/1667211911 conn(0x7f81fc0a04b0 msgr2=0x7f81fc0a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 -- 172.21.15.37:0/1667211911 shutdown_connections 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 -- 172.21.15.37:0/1667211911 wait complete. 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient: build_initial_monmap 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient: monmap: 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:55.100 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 Processor -- start 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 -- start start 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient: init 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81fc099f20 0x7f81fc1c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient(hunting): picked mon.noname-b con 0x7f81fc099f20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc13a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient(hunting): picked mon.noname-c con 0x7f81fc09a330 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 0x7f81fc11eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient(hunting): picked mon.noname-a con 0x7f81fc09ce00 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc122520 con 0x7f81fc09ce00 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc1c74d0 con 0x7f81fc099f20 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc070990 con 0x7f81fc09a330 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8203ecc700 10 monclient(hunting): authenticate will time out at 1095.574905s 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 0x7f81fc11eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 0x7f81fc11eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44096/0 (socket says 172.21.15.37:44096) 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 1 -- 172.21.15.37:0/1288656729 learned_addr learned my addr 172.21.15.37:0/1288656729 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 10 monclient(hunting): get_auth_request con 0x7f81fc09ce00 auth_method 0 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201467700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc13a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201467700 10 monclient(hunting): get_auth_request con 0x7f81fc09a330 auth_method 0 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201467700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201467700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201467700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201c68700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81fc099f20 0x7f81fc1c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201467700 1 -- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 msgr2=0x7f81fc13a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201467700 1 -- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 msgr2=0x7f81fc13a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201467700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc13a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 10 monclient(hunting): handle_auth_done global_id 34758 payload 824 2024-09-16T07:22:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201467700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc13a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 1 -- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 msgr2=0x7f81fc13a150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc13a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 1 -- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81fc099f20 msgr2=0x7f81fc1c6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81fc099f20 0x7f81fc1c6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 10 monclient: _finish_hunting 0 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 1 monclient: found mon.noname-a 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 1 -- 172.21.15.37:0/1288656729 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f80094b0 con 0x7f81fc09ce00 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201c68700 10 monclient: get_auth_request con 0x7f81fc099f20 auth_method 0 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8201c68700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81fc099f20 0x7f81fc1c6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.085+0000 7f8202469700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 0x7f81fc11eb80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f81f4007b00 tx=0x7f81f4007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 1 -- 172.21.15.37:0/1288656729 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f400da30 con 0x7f81fc09ce00 2024-09-16T07:22:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 10 monclient: dump: 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 10 monclient: _finish_auth 0 2024-09-16T07:22:55.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 10 monclient: _check_auth_tickets 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 5 monclient: authenticate success, global_id 34758 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 10 monclient: _renew_subs 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 1 -- 172.21.15.37:0/1288656729 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81fc0a32b0 con 0x7f81fc09ce00 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 10 monclient: _renew_subs 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 1 -- 172.21.15.37:0/1288656729 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f400f040 con 0x7f81fc09ce00 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 1 -- 172.21.15.37:0/1288656729 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81fc122ae0 con 0x7f81fc09ce00 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 1 -- 172.21.15.37:0/1288656729 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f400ddf0 con 0x7f81fc09ce00 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 10 monclient: dump: 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f37fe700 4 set_mon_vals no callback set 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 1 -- 172.21.15.37:0/1288656729 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f81f401e3b0 con 0x7f81fc09ce00 2024-09-16T07:22:55.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f81e0035600 0x7f81e0037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f81f27fc700 1 -- 172.21.15.37:0/1288656729 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f81f40479e0 con 0x7f81fc09ce00 2024-09-16T07:22:55.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8201c68700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f81e0035600 0x7f81e0037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8201c68700 10 monclient: get_auth_request con 0x7f81e0035600 auth_method 0 2024-09-16T07:22:55.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:55.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:55.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8203ecc700 1 -- 172.21.15.37:0/1288656729 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81fc120840 con 0x7f81fc09ce00 2024-09-16T07:22:55.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.089+0000 7f8201c68700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f81e0035600 0x7f81e0037ac0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f81f80090b0 tx=0x7f81f8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.097+0000 7f81f27fc700 1 -- 172.21.15.37:0/1288656729 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f81fc120840 con 0x7f81fc09ce00 2024-09-16T07:22:55.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.097+0000 7f81f27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:55.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.097+0000 7f81f27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:55.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.337+0000 7f8203ecc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:22:55.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.337+0000 7f8203ecc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:55.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.337+0000 7f8203ecc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.337+0000 7f8203ecc700 1 -- 172.21.15.37:0/1288656729 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f81fc122660 con 0x7f81fc09ce00 2024-09-16T07:22:55.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f81f27fc700 1 -- 172.21.15.37:0/1288656729 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f81fc122660 con 0x7f81fc09ce00 2024-09-16T07:22:55.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f81f27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:55.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f81f27fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:55.343 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 -- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f81e0035600 msgr2=0x7f81e0037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f81e0035600 0x7f81e0037ac0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f81f80090b0 tx=0x7f81f8008030 comp rx=0 tx=0).stop 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 10 monclient: shutdown 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 -- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 msgr2=0x7f81fc11eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 0x7f81fc11eb80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f81f4007b00 tx=0x7f81f4007ec0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 -- 172.21.15.37:0/1288656729 shutdown_connections 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc09ce00 0x7f81fc11eb80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f81e0035600 0x7f81e0037ac0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc09a330 0x7f81fc13a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 --2- 172.21.15.37:0/1288656729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81fc099f20 0x7f81fc1c6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 -- 172.21.15.37:0/1288656729 >> 172.21.15.37:0/1288656729 conn(0x7f81fc0a04b0 msgr2=0x7f81fc06f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 -- 172.21.15.37:0/1288656729 shutdown_connections 2024-09-16T07:22:55.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.341+0000 7f8203ecc700 1 -- 172.21.15.37:0/1288656729 wait complete. 2024-09-16T07:22:55.357 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T07:22:55.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T07:22:55.358 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T07:22:55.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient: get_monmap_and_config 2024-09-16T07:22:55.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient: build_initial_monmap 2024-09-16T07:22:55.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient: monmap: 2024-09-16T07:22:55.468 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:55.468 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.468 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:55.468558+0000 2024-09-16T07:22:55.468 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:55.468558+0000 2024-09-16T07:22:55.468 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:55.469 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.469 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:55.469 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:55.469 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:55.469 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:55.469 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:55.469 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 1 Processor -- start 2024-09-16T07:22:55.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 1 -- start start 2024-09-16T07:22:55.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient: init 2024-09-16T07:22:55.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:55.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T07:22:55.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a0e90 0x7f52a40a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient(hunting): picked mon.noname-b con 0x7f52a40a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f52a40a1270 0x7f52a40a2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient(hunting): picked mon.noname-c con 0x7f52a40a1270 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52a40a2b80 addr v1:172.21.15.143:6789/0 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a40a36c0 con 0x7f52a40a2b80 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a406cf50 con 0x7f52a40a1270 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a409f810 con 0x7f52a40a0e90 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a96be700 10 monclient(hunting): authenticate will time out at 1095.955473s 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a3fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a0e90 0x7f52a40a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a3fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a0e90 0x7f52a40a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50532/0 (socket says 172.21.15.37:50532) 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a3fff700 1 -- 172.21.15.37:0/4042423429 learned_addr learned my addr 172.21.15.37:0/4042423429 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a3fff700 10 monclient(hunting): get_auth_request con 0x7f52a40a0e90 auth_method 0 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a37fe700 1 --2- 172.21.15.37:0/4042423429 >> v2:172.21.15.143:3300/0 conn(0x7f52a40a1270 0x7f52a40a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a37fe700 10 monclient(hunting): get_auth_request con 0x7f52a40a1270 auth_method 0 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a3fff700 1 -- 172.21.15.37:0/4042423429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a0e90 msgr2=0x7f52a40a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:22:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a3fff700 1 -- 172.21.15.37:0/4042423429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a0e90 msgr2=0x7f52a40a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.465+0000 7f52a3fff700 1 --2- 172.21.15.37:0/4042423429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a0e90 0x7f52a40a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a3fff700 1 --2- 172.21.15.37:0/4042423429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a0e90 0x7f52a40a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 10 monclient(hunting): handle_auth_done global_id 5659 payload 274 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 1 -- 172.21.15.37:0/4042423429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a0e90 msgr2=0x7f52a40a04d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 1 --2- 172.21.15.37:0/4042423429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a0e90 0x7f52a40a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 1 -- 172.21.15.37:0/4042423429 >> v1:172.21.15.143:6789/0 conn(0x7f52a40a2b80 legacy=0x7f52a40a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 10 monclient: _finish_hunting 0 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 1 monclient: found mon.noname-c 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 1 -- 172.21.15.37:0/4042423429 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a409f250 con 0x7f52a40a1270 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a37fe700 1 --2- 172.21.15.37:0/4042423429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a40a1270 0x7f52a40a2640 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5298009850 tx=0x7f5298009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e940f0a1a7f198e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 1 -- 172.21.15.37:0/4042423429 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298011830 con 0x7f52a40a1270 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 10 monclient: dump: 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 10 monclient: _finish_auth 0 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 5 monclient: authenticate success, global_id 5659 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 1 -- 172.21.15.37:0/4042423429 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5298011990 con 0x7f52a40a1270 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:55.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 4 set_mon_vals no callback set 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 1 -- 172.21.15.37:0/4042423429 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298011d20 con 0x7f52a40a1270 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a2ffd700 10 monclient: dump: 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.480 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 10 monclient: shutdown 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 1 -- 172.21.15.37:0/4042423429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a40a1270 msgr2=0x7f52a40a2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 1 --2- 172.21.15.37:0/4042423429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a40a1270 0x7f52a40a2640 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5298009850 tx=0x7f5298009b60 comp rx=0 tx=0).stop 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 1 -- 172.21.15.37:0/4042423429 shutdown_connections 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 1 --2- 172.21.15.37:0/4042423429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a40a1270 0x7f52a40a2640 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 1 --2- 172.21.15.37:0/4042423429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a0e90 0x7f52a40a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 1 -- 172.21.15.37:0/4042423429 >> 172.21.15.37:0/4042423429 conn(0x7f52a40a4740 msgr2=0x7f52a40a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 1 -- 172.21.15.37:0/4042423429 shutdown_connections 2024-09-16T07:22:55.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.469+0000 7f52a96be700 1 -- 172.21.15.37:0/4042423429 wait complete. 2024-09-16T07:22:55.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient: build_initial_monmap 2024-09-16T07:22:55.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient: monmap: 2024-09-16T07:22:55.488 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:55.488 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:55.488 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:55.488 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:55.488 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:55.488 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:55.488 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:55.488 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 1 Processor -- start 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 1 -- start start 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient: init 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a40a0e90 0x7f52a413b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient(hunting): picked mon.noname-b con 0x7f52a40a0e90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a40a1270 0x7f52a413bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient(hunting): picked mon.noname-a con 0x7f52a40a1270 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a2b80 0x7f52a413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient(hunting): picked mon.noname-c con 0x7f52a40a2b80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a406d780 con 0x7f52a40a1270 2024-09-16T07:22:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4125410 con 0x7f52a40a0e90 2024-09-16T07:22:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a409df80 con 0x7f52a40a2b80 2024-09-16T07:22:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient(hunting): authenticate will time out at 1095.962332s 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a2b80 0x7f52a413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a40a0e90 0x7f52a413b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a40a1270 0x7f52a413bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a2b80 0x7f52a413c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50546/0 (socket says 172.21.15.37:50546) 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 1 -- 172.21.15.37:0/443750612 learned_addr learned my addr 172.21.15.37:0/443750612 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 10 monclient(hunting): get_auth_request con 0x7f52a40a2b80 auth_method 0 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 10 monclient(hunting): get_auth_request con 0x7f52a40a1270 auth_method 0 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a3fff700 10 monclient(hunting): get_auth_request con 0x7f52a40a0e90 auth_method 0 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 1 -- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a2b80 msgr2=0x7f52a413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 1 -- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a2b80 msgr2=0x7f52a413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a2b80 0x7f52a413c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a8ebd700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a2b80 0x7f52a413c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:55.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 10 monclient(hunting): handle_auth_done global_id 34761 payload 824 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 1 -- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a2b80 msgr2=0x7f52a413c2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52a40a2b80 0x7f52a413c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 1 -- 172.21.15.37:0/443750612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a40a0e90 msgr2=0x7f52a413b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a40a0e90 0x7f52a413b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 10 monclient: _finish_hunting 0 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 1 monclient: found mon.noname-a 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 1 -- 172.21.15.37:0/443750612 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52980094b0 con 0x7f52a40a1270 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a37fe700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a40a1270 0x7f52a413bdb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5298016000 tx=0x7f5298004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298004480 con 0x7f52a40a1270 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a0ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a0ff9700 10 monclient: dump: 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a0ff9700 10 monclient: _finish_auth 0 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a0ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:55.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 5 monclient: authenticate success, global_id 34761 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529801f070 con 0x7f52a40a1270 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient: _renew_subs 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a96be700 1 -- 172.21.15.37:0/443750612 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a406c010 con 0x7f52a40a1270 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.473+0000 7f52a0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298003900 con 0x7f52a40a1270 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a27fc700 4 set_mon_vals no callback set 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 10 monclient: dump: 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.495 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a96be700 10 monclient: _renew_subs 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a96be700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a96be700 1 -- 172.21.15.37:0/443750612 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a4126ba0 con 0x7f52a40a1270 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f5298021030 con 0x7f52a40a1270 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5290035600 0x7f5290037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f5298049190 con 0x7f52a40a1270 2024-09-16T07:22:55.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a3fff700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5290035600 0x7f5290037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a3fff700 10 monclient: get_auth_request con 0x7f5290035600 auth_method 0 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a96be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a96be700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4123dc0 0x7f52a4061580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a96be700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a96be700 1 -- 172.21.15.37:0/443750612 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4126d30 con 0x7f52a4123dc0 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4123dc0 0x7f52a4061580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 10 monclient: get_auth_request con 0x7f52a4123dc0 auth_method 0 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:55.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 10 monclient(hunting): handle_auth_done global_id 34764 payload 824 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 1 -- 172.21.15.37:0/443750612 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52a40a1670 con 0x7f52a4123dc0 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a3fff700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5290035600 0x7f5290037ac0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f52940091a0 tx=0x7f5294008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a8ebd700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4123dc0 0x7f52a4061580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f529c0012e0 tx=0x7f529c00c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c0020b0 con 0x7f52a4123dc0 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 10 monclient: dump: 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.498 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f529c003c30 con 0x7f52a4123dc0 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4123dc0 msgr2=0x7f52a4061580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4123dc0 0x7f52a4061580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f529c0012e0 tx=0x7f529c00c3c0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4123dc0 msgr2=0x7f52a4061580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:55.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.477+0000 7f52a0ff9700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4123dc0 0x7f52a4061580 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:22:55.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4061ac0 0x7f52a402b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 -- 172.21.15.37:0/443750612 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4067400 con 0x7f52a4061ac0 2024-09-16T07:22:55.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4061ac0 0x7f52a402b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 10 monclient: get_auth_request con 0x7f52a4061ac0 auth_method 0 2024-09-16T07:22:55.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 10 monclient(hunting): handle_auth_done global_id 34767 payload 824 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 1 -- 172.21.15.37:0/443750612 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f52a409f5d0 con 0x7f52a4061ac0 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4061ac0 0x7f52a402b6a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f529c0012e0 tx=0x7f529c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c022070 con 0x7f52a4061ac0 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 10 monclient: dump: 2024-09-16T07:22:55.502 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4619 (secure 0 0 0) 0x7f529c01de30 con 0x7f52a4061ac0 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:55.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4061ac0 msgr2=0x7f52a402b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4061ac0 0x7f52a402b6a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f529c0012e0 tx=0x7f529c0014a0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 1 -- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4061ac0 msgr2=0x7f52a402b6a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:55.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a0ff9700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a4061ac0 0x7f52a402b6a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.504 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:22:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:22:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:22:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:22:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 22, 2024-09-16T07:22:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:22:55.504 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:55.505 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:55.506 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:22:55.507 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:22:55.508 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.509 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.511 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:22:55.512 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:22:55.513 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:55.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:55.515 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:55.516 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:22:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:55.517 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:22:55.518 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 -- 172.21.15.37:0/443750612 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5290035600 msgr2=0x7f5290037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f5290035600 0x7f5290037ac0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f52940091a0 tx=0x7f5294008040 comp rx=0 tx=0).stop 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 10 monclient: shutdown 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a8ebd700 1 -- 172.21.15.37:0/443750612 reap_dead start 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 -- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a40a1270 msgr2=0x7f52a413bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a40a1270 0x7f52a413bdb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5298016000 tx=0x7f5298004770 comp rx=0 tx=0).stop 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 -- 172.21.15.37:0/443750612 shutdown_connections 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 --2- 172.21.15.37:0/443750612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52a40a1270 0x7f52a413bdb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 -- 172.21.15.37:0/443750612 >> 172.21.15.37:0/443750612 conn(0x7f52a40a4740 msgr2=0x7f52a409e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:55.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 -- 172.21.15.37:0/443750612 shutdown_connections 2024-09-16T07:22:55.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.489+0000 7f52a96be700 1 -- 172.21.15.37:0/443750612 wait complete. 2024-09-16T07:22:55.521 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:22:55.521 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T07:22:55.521 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T07:22:55.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.641+0000 7f2937fff700 10 monclient: get_monmap_and_config 2024-09-16T07:22:55.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.641+0000 7f2937fff700 10 monclient: build_initial_monmap 2024-09-16T07:22:55.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.641+0000 7f2937fff700 10 monclient: monmap: 2024-09-16T07:22:55.646 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:55.646 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.646 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:55.646199+0000 2024-09-16T07:22:55.646 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:55.646199+0000 2024-09-16T07:22:55.646 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:55.646 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.646 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:55.647 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:55.647 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:55.647 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:55.647 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:55.647 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 1 Processor -- start 2024-09-16T07:22:55.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 1 -- start start 2024-09-16T07:22:55.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 10 monclient: init 2024-09-16T07:22:55.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:55.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T07:22:55.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f29380a2640 0x7f29380a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 10 monclient(hunting): picked mon.noname-a con 0x7f29380a2640 addr v2:172.21.15.37:3300/0 2024-09-16T07:22:55.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f29380a2f60 addr v1:172.21.15.143:6789/0 2024-09-16T07:22:55.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2938135b10 addr v1:172.21.15.37:6789/0 2024-09-16T07:22:55.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29380a3770 con 0x7f2938135b10 2024-09-16T07:22:55.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293806cbe0 con 0x7f29380a2f60 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293809fb80 con 0x7f29380a2640 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 10 monclient(hunting): authenticate will time out at 1096.133270s 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f29377fe700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2938135b10 0x7f293813d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47230/0 (socket says 172.21.15.37:47230) 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f29380a2640 0x7f29380a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f29377fe700 1 -- 172.21.15.37:0/777620259 learned_addr learned my addr 172.21.15.37:0/777620259 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 10 monclient(hunting): get_auth_request con 0x7f29380a2640 auth_method 0 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 10 monclient(hunting): handle_auth_done global_id 34770 payload 274 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 1 -- 172.21.15.37:0/777620259 >> v1:172.21.15.143:6789/0 conn(0x7f29380a2f60 legacy=0x7f2938135400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 1 -- 172.21.15.37:0/777620259 >> v1:172.21.15.37:6789/0 conn(0x7f2938135b10 legacy=0x7f293813d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 10 monclient: _finish_hunting 0 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 1 monclient: found mon.noname-a 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 1 -- 172.21.15.37:0/777620259 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29380a1f80 con 0x7f29380a2640 2024-09-16T07:22:55.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2936ffd700 1 --2- 172.21.15.37:0/777620259 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29380a2640 0x7f29380a2a20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2928009850 tx=0x7f2928009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00572fb0b6d657b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.652 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:55.652 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27273]: starting. 2024-09-16T07:22:55.652 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27273]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27235 2024-09-16T07:22:55.652 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27273]: shutting down. 2024-09-16T07:22:55.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 1 -- 172.21.15.37:0/777620259 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928011830 con 0x7f29380a2640 2024-09-16T07:22:55.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:55.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 10 monclient: dump: 2024-09-16T07:22:55.654 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.654 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.654 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.654 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.654 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.654 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.655 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.655 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.655 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.655 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 10 monclient: _finish_auth 0 2024-09-16T07:22:55.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:22:55.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:55.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 5 monclient: authenticate success, global_id 34770 2024-09-16T07:22:55.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 1 -- 172.21.15.37:0/777620259 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2928011990 con 0x7f29380a2640 2024-09-16T07:22:55.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:55.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:55.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:55.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2937fff700 4 set_mon_vals no callback set 2024-09-16T07:22:55.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 1 -- 172.21.15.37:0/777620259 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928011d20 con 0x7f29380a2640 2024-09-16T07:22:55.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:55.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.645+0000 7f2935ffb700 10 monclient: dump: 2024-09-16T07:22:55.656 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.656 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.657 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.657 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.657 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.657 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.657 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.657 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.657 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.657 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 10 monclient: shutdown 2024-09-16T07:22:55.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 1 -- 172.21.15.37:0/777620259 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29380a2640 msgr2=0x7f29380a2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 1 --2- 172.21.15.37:0/777620259 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29380a2640 0x7f29380a2a20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2928009850 tx=0x7f2928009b60 comp rx=0 tx=0).stop 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 1 -- 172.21.15.37:0/777620259 shutdown_connections 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 1 --2- 172.21.15.37:0/777620259 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29380a2640 0x7f29380a2a20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 1 -- 172.21.15.37:0/777620259 >> 172.21.15.37:0/777620259 conn(0x7f29380a8270 msgr2=0x7f29380a4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 1 -- 172.21.15.37:0/777620259 shutdown_connections 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 1 -- 172.21.15.37:0/777620259 wait complete. 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 10 monclient: build_initial_monmap 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 10 monclient: monmap: 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:55.658 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:55.659 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:55.659 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:55.659 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:55.659 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:55.659 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:55.659 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 1 Processor -- start 2024-09-16T07:22:55.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 1 -- start start 2024-09-16T07:22:55.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.649+0000 7f2937fff700 10 monclient: init 2024-09-16T07:22:55.661 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:55.661 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:22:55.661 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:22:55.661 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:22:55.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:55.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:22:55.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29380a2640 0x7f2938134660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient(hunting): picked mon.noname-c con 0x7f29380a2640 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:55.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29380a2f60 0x7f2938134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient(hunting): picked mon.noname-a con 0x7f29380a2f60 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2938135b10 0x7f29381221d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient(hunting): picked mon.noname-b con 0x7f2938135b10 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2938125b50 con 0x7f29380a2f60 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2938125da0 con 0x7f2938135b10 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29381cad50 con 0x7f29380a2640 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient(hunting): authenticate will time out at 1096.140268s 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29380a2640 0x7f2938134660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29380a2640 0x7f2938134660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50562/0 (socket says 172.21.15.37:50562) 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 learned_addr learned my addr 172.21.15.37:0/1746866829 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 10 monclient(hunting): get_auth_request con 0x7f29380a2640 auth_method 0 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29380a2f60 0x7f2938134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 10 monclient(hunting): get_auth_request con 0x7f29380a2f60 auth_method 0 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29380a2640 msgr2=0x7f2938134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29380a2640 msgr2=0x7f2938134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29380a2640 0x7f2938134660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2938135b10 0x7f29381221d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29380a2640 0x7f2938134660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 10 monclient(hunting): get_auth_request con 0x7f2938135b10 auth_method 0 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:55.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 10 monclient(hunting): handle_auth_done global_id 34773 payload 824 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29380a2640 msgr2=0x7f2938134660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29380a2640 0x7f2938134660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2938135b10 msgr2=0x7f29381221d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2938135b10 0x7f29381221d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 10 monclient: _finish_hunting 0 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 1 monclient: found mon.noname-a 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 1 -- 172.21.15.37:0/1746866829 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280094b0 con 0x7f29380a2f60 2024-09-16T07:22:55.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29367fc700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29380a2f60 0x7f2938134ba0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f293000cbe0 tx=0x7f293000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930003c20 con 0x7f29380a2f60 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 10 monclient: dump: 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.670 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 10 monclient: _finish_auth 0 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 10 monclient: _check_auth_tickets 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 5 monclient: authenticate success, global_id 34773 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2930003d80 con 0x7f29380a2f60 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient: _renew_subs 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 1 -- 172.21.15.37:0/1746866829 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2938135400 con 0x7f29380a2f60 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293000ab50 con 0x7f29380a2f60 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:55.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 10 monclient: dump: 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29357fa700 4 set_mon_vals no callback set 2024-09-16T07:22:55.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient: _renew_subs 2024-09-16T07:22:55.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:55.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 1 -- 172.21.15.37:0/1746866829 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2938126080 con 0x7f29380a2f60 2024-09-16T07:22:55.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:55.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:55.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2937fff700 1 -- 172.21.15.37:0/1746866829 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2938067450 con 0x7f2938122710 2024-09-16T07:22:55.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 10 monclient: get_auth_request con 0x7f2938122710 auth_method 0 2024-09-16T07:22:55.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f293000acb0 con 0x7f29380a2f60 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 msgr2=0x7f2938122af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 msgr2=0x7f2938122af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2924035600 0x7f2924037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f2930047a90 con 0x7f29380a2f60 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2924035600 0x7f2924037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 10 monclient: get_auth_request con 0x7f2924035600 auth_method 0 2024-09-16T07:22:55.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.653+0000 7f29377fe700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2924035600 0x7f2924037ac0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f292c005930 tx=0x7f292c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.857+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.857+0000 7f2936ffd700 10 monclient: get_auth_request con 0x7f2938122710 auth_method 0 2024-09-16T07:22:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.857+0000 7f2936ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.857+0000 7f2936ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.857+0000 7f2936ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:22:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.857+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 msgr2=0x7f2938122af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:22:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.857+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 msgr2=0x7f2938122af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:22:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.857+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:55.857+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:22:56.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:56.261+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:56.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:56.261+0000 7f2936ffd700 10 monclient: get_auth_request con 0x7f2938122710 auth_method 0 2024-09-16T07:22:56.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:56.261+0000 7f2936ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:56.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:56.261+0000 7f2936ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:56.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:56.261+0000 7f2936ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:22:56.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:56.261+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 msgr2=0x7f2938122af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:22:56.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:56.261+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 msgr2=0x7f2938122af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:22:56.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:56.261+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:22:56.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:56.261+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:22:57.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 10 monclient: get_auth_request con 0x7f2938122710 auth_method 0 2024-09-16T07:22:57.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:22:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 10 monclient(hunting): handle_auth_done global_id 44099 payload 824 2024-09-16T07:22:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2938125f10 con 0x7f2938122710 2024-09-16T07:22:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2928000c00 tx=0x7f2928004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928020050 con 0x7f2938122710 2024-09-16T07:22:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:22:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 10 monclient: dump: 2024-09-16T07:22:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.074 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f2928021040 con 0x7f2938122710 2024-09-16T07:22:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 msgr2=0x7f2938122af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2928000c00 tx=0x7f2928004770 comp rx=0 tx=0).stop 2024-09-16T07:22:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 msgr2=0x7f2938122af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.065+0000 7f291ffff700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2938122710 0x7f2938122af0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2928000c00 tx=0x7f2928004770 comp rx=0 tx=0).stop 2024-09-16T07:22:57.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2937fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:22:57.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2937fff700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f293811e290 0x7f293811e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2937fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2937fff700 1 -- 172.21.15.37:0/1746866829 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29380a3370 con 0x7f293811e290 2024-09-16T07:22:57.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f293811e290 0x7f293811e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 10 monclient: get_auth_request con 0x7f293811e290 auth_method 0 2024-09-16T07:22:57.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 10 monclient(hunting): handle_auth_done global_id 44102 payload 824 2024-09-16T07:22:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 1 -- 172.21.15.37:0/1746866829 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f293806ce20 con 0x7f293811e290 2024-09-16T07:22:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2936ffd700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f293811e290 0x7f293811e6a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2928009ba0 tx=0x7f2928009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292802ed70 con 0x7f293811e290 2024-09-16T07:22:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:22:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 10 monclient: dump: 2024-09-16T07:22:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.089 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4453 (secure 0 0 0) 0x7f2928026460 con 0x7f293811e290 2024-09-16T07:22:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:22:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f293811e290 msgr2=0x7f293811e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f293811e290 0x7f293811e6a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2928009ba0 tx=0x7f2928009240 comp rx=0 tx=0).stop 2024-09-16T07:22:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f293811e290 msgr2=0x7f293811e6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f291ffff700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f293811e290 0x7f293811e6a0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2928009ba0 tx=0x7f2928009240 comp rx=0 tx=0).stop 2024-09-16T07:22:57.091 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:22:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:22:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:22:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:22:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 22, 2024-09-16T07:22:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:22:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:22:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:22:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:22:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-09-16T07:22:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:22:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:22:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:22:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:22:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:22:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:22:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:22:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:22:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:22:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:22:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:22:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:22:57.105 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:22:57.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2937fff700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2924035600 msgr2=0x7f2924037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2937fff700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2924035600 0x7f2924037ac0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f292c005930 tx=0x7f292c009040 comp rx=0 tx=0).stop 2024-09-16T07:22:57.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2937fff700 10 monclient: shutdown 2024-09-16T07:22:57.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2937fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:57.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2937fff700 1 -- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29380a2f60 msgr2=0x7f2938134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.077+0000 7f2937fff700 1 --2- 172.21.15.37:0/1746866829 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29380a2f60 0x7f2938134ba0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f293000cbe0 tx=0x7f293000cef0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.081+0000 7f29377fe700 1 -- 172.21.15.37:0/1746866829 reap_dead start 2024-09-16T07:22:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.081+0000 7f29377fe700 1 -- 172.21.15.37:0/1746866829 reap_dead start 2024-09-16T07:22:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.081+0000 7f2937fff700 1 -- 172.21.15.37:0/1746866829 shutdown_connections 2024-09-16T07:22:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.081+0000 7f2937fff700 1 -- 172.21.15.37:0/1746866829 >> 172.21.15.37:0/1746866829 conn(0x7f29380a8270 msgr2=0x7f29380a64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.081+0000 7f2937fff700 1 -- 172.21.15.37:0/1746866829 shutdown_connections 2024-09-16T07:22:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.081+0000 7f2937fff700 1 -- 172.21.15.37:0/1746866829 wait complete. 2024-09-16T07:22:57.109 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:22:57.109 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T07:22:57.109 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T07:22:57.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.229+0000 7fe48d3e3700 10 monclient: get_monmap_and_config 2024-09-16T07:22:57.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.229+0000 7fe48d3e3700 10 monclient: build_initial_monmap 2024-09-16T07:22:57.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.229+0000 7fe48d3e3700 10 monclient: monmap: 2024-09-16T07:22:57.234 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:57.234 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.234 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:57.234222+0000 2024-09-16T07:22:57.234 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:57.234222+0000 2024-09-16T07:22:57.234 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:57.234 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.234 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:57.235 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:57.235 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:57.235 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:57.235 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:57.235 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.229+0000 7fe48d3e3700 1 Processor -- start 2024-09-16T07:22:57.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.229+0000 7fe48d3e3700 1 -- start start 2024-09-16T07:22:57.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.229+0000 7fe48d3e3700 10 monclient: init 2024-09-16T07:22:57.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:57.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T07:22:57.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe4880a1090 0x7fe4880a0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 10 monclient(hunting): picked mon.noname-c con 0x7fe4880a1090 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:57.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4880a1470 addr v1:172.21.15.37:6789/0 2024-09-16T07:22:57.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4880a1850 addr v1:172.21.15.143:6789/0 2024-09-16T07:22:57.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48809fb30 con 0x7fe4880a1470 2024-09-16T07:22:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48809f850 con 0x7fe4880a1850 2024-09-16T07:22:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4880a1ca0 con 0x7fe4880a1090 2024-09-16T07:22:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 10 monclient(hunting): authenticate will time out at 1097.720382s 2024-09-16T07:22:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe4867fc700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe4880a1470 0x7fe4880a2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47246/0 (socket says 172.21.15.37:47246) 2024-09-16T07:22:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe4867fc700 1 -- 172.21.15.37:0/1874018949 learned_addr learned my addr 172.21.15.37:0/1874018949 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:22:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 1 --2- 172.21.15.37:0/1874018949 >> v2:172.21.15.143:3300/0 conn(0x7fe4880a1090 0x7fe4880a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 10 monclient(hunting): get_auth_request con 0x7fe4880a1090 auth_method 0 2024-09-16T07:22:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 1 -- 172.21.15.37:0/1874018949 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515047172 0 0) 0x7fe48809fb30 con 0x7fe4880a1470 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient(hunting): _negotiate 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient(hunting): my global_id is 34788 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 1 -- 172.21.15.37:0/1874018949 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4700035d0 con 0x7fe4880a1470 2024-09-16T07:22:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 1 -- 172.21.15.37:0/1874018949 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700956259 0 0) 0x7fe48809f850 con 0x7fe4880a1850 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient(hunting): _negotiate 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient(hunting): my global_id is 5686 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 1 -- 172.21.15.37:0/1874018949 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe48809fb30 con 0x7fe4880a1850 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 10 monclient(hunting): handle_auth_done global_id 5683 payload 274 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 1 -- 172.21.15.37:0/1874018949 >> v1:172.21.15.143:6789/0 conn(0x7fe4880a1850 legacy=0x7fe4880a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 1 -- 172.21.15.37:0/1874018949 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2031668624 0 0) 0x7fe4700035d0 con 0x7fe4880a1470 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 1 -- 172.21.15.37:0/1874018949 >> v1:172.21.15.37:6789/0 conn(0x7fe4880a1470 legacy=0x7fe4880a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 10 monclient: _finish_hunting 0 2024-09-16T07:22:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 1 monclient: found mon.noname-c 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 1 -- 172.21.15.37:0/1874018949 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4880a3570 con 0x7fe4880a1090 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe486ffd700 1 --2- 172.21.15.37:0/1874018949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4880a1090 0x7fe4880a0840 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009850 tx=0x7fe47c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15f05351521b2237 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 1 -- 172.21.15.37:0/1874018949 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478001670 con 0x7fe4880a1090 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient: dump: 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient: _finish_auth 0 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 5 monclient: authenticate success, global_id 5683 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 1 -- 172.21.15.37:0/1874018949 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c004030 con 0x7fe4880a1090 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 4 set_mon_vals no callback set 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 1 -- 172.21.15.37:0/1874018949 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c0043a0 con 0x7fe4880a1090 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe485ffb700 10 monclient: dump: 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.246 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 10 monclient: shutdown 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 1 -- 172.21.15.37:0/1874018949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4880a1090 msgr2=0x7fe4880a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 1 --2- 172.21.15.37:0/1874018949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4880a1090 0x7fe4880a0840 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009850 tx=0x7fe47c009b60 comp rx=0 tx=0).stop 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 1 -- 172.21.15.37:0/1874018949 shutdown_connections 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 1 --2- 172.21.15.37:0/1874018949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4880a1090 0x7fe4880a0840 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 1 -- 172.21.15.37:0/1874018949 >> 172.21.15.37:0/1874018949 conn(0x7fe4880a8730 msgr2=0x7fe4880a4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.233+0000 7fe48d3e3700 1 -- 172.21.15.37:0/1874018949 shutdown_connections 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 1 -- 172.21.15.37:0/1874018949 wait complete. 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient: build_initial_monmap 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient: monmap: 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:57.248 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:57.249 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:57.249 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:57.249 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:57.249 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:57.249 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:57.249 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:57.249 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 1 Processor -- start 2024-09-16T07:22:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 1 -- start start 2024-09-16T07:22:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient: init 2024-09-16T07:22:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:22:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4880a1090 0x7fe4881c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient(hunting): picked mon.noname-b con 0x7fe4880a1090 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4880a1470 0x7fe4881c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient(hunting): picked mon.noname-c con 0x7fe4880a1470 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4880a1850 0x7fe4881c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient(hunting): picked mon.noname-a con 0x7fe4880a1850 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4881a8f20 con 0x7fe4880a1850 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880a7670 con 0x7fe4880a1090 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48809e000 con 0x7fe4880a1470 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe48d3e3700 10 monclient(hunting): authenticate will time out at 1097.726857s 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4880a1850 0x7fe4881c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4880a1850 0x7fe4881c6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44120/0 (socket says 172.21.15.37:44120) 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 1 -- 172.21.15.37:0/2405575582 learned_addr learned my addr 172.21.15.37:0/2405575582 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 10 monclient(hunting): get_auth_request con 0x7fe4880a1850 auth_method 0 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4880a1470 0x7fe4881c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 10 monclient(hunting): get_auth_request con 0x7fe4880a1470 auth_method 0 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe486ffd700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4880a1090 0x7fe4881c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe486ffd700 10 monclient(hunting): get_auth_request con 0x7fe4880a1090 auth_method 0 2024-09-16T07:22:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe486ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe486ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe486ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 10 monclient(hunting): handle_auth_done global_id 44105 payload 824 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 1 -- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4880a1090 msgr2=0x7fe4881c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4880a1090 0x7fe4881c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 1 -- 172.21.15.37:0/2405575582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4880a1850 msgr2=0x7fe4881c6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4880a1850 0x7fe4881c6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 10 monclient: _finish_hunting 0 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 1 monclient: found mon.noname-c 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4867fc700 1 -- 172.21.15.37:0/2405575582 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47c0094b0 con 0x7fe4880a1470 2024-09-16T07:22:57.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.237+0000 7fe4877fe700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4880a1850 0x7fe4881c6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe4867fc700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4880a1470 0x7fe4881c3ff0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe47800c550 tx=0x7fe47800c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478004ca0 con 0x7fe4880a1470 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: dump: 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: _finish_auth 0 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe478004e00 con 0x7fe4880a1470 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 5 monclient: authenticate success, global_id 44105 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 10 monclient: _renew_subs 2024-09-16T07:22:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:22:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 1 -- 172.21.15.37:0/2405575582 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48809f0b0 con 0x7fe4880a1470 2024-09-16T07:22:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4780072f0 con 0x7fe4880a1470 2024-09-16T07:22:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:22:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: dump: 2024-09-16T07:22:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe4857fa700 4 set_mon_vals no callback set 2024-09-16T07:22:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 10 monclient: _renew_subs 2024-09-16T07:22:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:22:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 1 -- 172.21.15.37:0/2405575582 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4881ab0d0 con 0x7fe4880a1470 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802b6a0 0x7fe48802ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe48d3e3700 1 -- 172.21.15.37:0/2405575582 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488067630 con 0x7fe48802b6a0 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fe478007450 con 0x7fe4880a1470 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802b6a0 0x7fe48802ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 10 monclient: get_auth_request con 0x7fe48802b6a0 auth_method 0 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe474035600 0x7fe474037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 4738+0+0 (secure 0 0 0) 0x7fe478047860 con 0x7fe4880a1470 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 10 monclient(hunting): handle_auth_done global_id 5692 payload 824 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe4877fe700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe474035600 0x7fe474037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 1 -- 172.21.15.37:0/2405575582 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe48806d370 con 0x7fe48802b6a0 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe4877fe700 10 monclient: get_auth_request con 0x7fe474035600 auth_method 0 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe486ffd700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802b6a0 0x7fe48802ba80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009850 tx=0x7fe47c004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c020070 con 0x7fe48802b6a0 2024-09-16T07:22:57.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe46f7fe700 10 monclient: dump: 2024-09-16T07:22:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.268 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.241+0000 7fe4877fe700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe474035600 0x7fe474037ac0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe480001560 tx=0x7fe480009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.245+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fe47c003ab0 con 0x7fe48802b6a0 2024-09-16T07:22:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.245+0000 7fe46f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.245+0000 7fe46f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.245+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802b6a0 msgr2=0x7fe48802ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.245+0000 7fe46f7fe700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802b6a0 0x7fe48802ba80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009850 tx=0x7fe47c004770 comp rx=0 tx=0).stop 2024-09-16T07:22:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.245+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802b6a0 msgr2=0x7fe48802ba80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.245+0000 7fe46f7fe700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802b6a0 0x7fe48802ba80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe48d3e3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:22:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe48d3e3700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802bfc0 0x7fe48802c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe48d3e3700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe48d3e3700 1 -- 172.21.15.37:0/2405575582 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48809f5b0 con 0x7fe48802bfc0 2024-09-16T07:22:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802bfc0 0x7fe48802c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 10 monclient: get_auth_request con 0x7fe48802bfc0 auth_method 0 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 10 monclient(hunting): handle_auth_done global_id 5695 payload 824 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 1 -- 172.21.15.37:0/2405575582 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe48806ca00 con 0x7fe48802bfc0 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe486ffd700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802bfc0 0x7fe48802c3d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe47c024000 tx=0x7fe47c011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c02cb30 con 0x7fe48802bfc0 2024-09-16T07:22:57.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe46f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe46f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:57.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.253+0000 7fe46f7fe700 10 monclient: dump: 2024-09-16T07:22:57.272 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.273 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.273 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.273 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.273 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.273 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.273 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.273 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.273 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.273 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4787 (secure 0 0 0) 0x7fe47c02cc90 con 0x7fe48802bfc0 2024-09-16T07:22:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe46f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:22:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe46f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802bfc0 msgr2=0x7fe48802c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe46f7fe700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802bfc0 0x7fe48802c3d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe47c024000 tx=0x7fe47c011000 comp rx=0 tx=0).stop 2024-09-16T07:22:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe46f7fe700 1 -- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802bfc0 msgr2=0x7fe48802c3d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe46f7fe700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe48802bfc0 0x7fe48802c3d0 secure :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe47c024000 tx=0x7fe47c011000 comp rx=0 tx=0).stop 2024-09-16T07:22:57.274 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:22:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:22:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:22:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-09-16T07:22:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 22, 2024-09-16T07:22:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:22:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:22:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:22:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:22:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-09-16T07:22:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:22:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:22:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:22:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:22:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:22:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.285 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.286 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.287 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:22:57.288 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.289 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:57.289 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:57.289 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:22:57.289 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.289 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:57.289 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:22:57.289 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.289 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:57.293 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:57.294 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:57.294 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:22:57.294 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:22:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe48d3e3700 1 -- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe474035600 msgr2=0x7fe474037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe48d3e3700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fe474035600 0x7fe474037ac0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe480001560 tx=0x7fe480009040 comp rx=0 tx=0).stop 2024-09-16T07:22:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe48d3e3700 10 monclient: shutdown 2024-09-16T07:22:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe48d3e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe48d3e3700 1 -- 172.21.15.37:0/2405575582 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4880a1470 msgr2=0x7fe4881c3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe48d3e3700 1 --2- 172.21.15.37:0/2405575582 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4880a1470 0x7fe4881c3ff0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe47800c550 tx=0x7fe47800c860 comp rx=0 tx=0).stop 2024-09-16T07:22:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe4877fe700 1 -- 172.21.15.37:0/2405575582 reap_dead start 2024-09-16T07:22:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe4877fe700 1 -- 172.21.15.37:0/2405575582 reap_dead start 2024-09-16T07:22:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe48d3e3700 1 -- 172.21.15.37:0/2405575582 shutdown_connections 2024-09-16T07:22:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe48d3e3700 1 -- 172.21.15.37:0/2405575582 >> 172.21.15.37:0/2405575582 conn(0x7fe4880a8730 msgr2=0x7fe4880a6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe48d3e3700 1 -- 172.21.15.37:0/2405575582 shutdown_connections 2024-09-16T07:22:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.257+0000 7fe48d3e3700 1 -- 172.21.15.37:0/2405575582 wait complete. 2024-09-16T07:22:57.297 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:22:57.297 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T07:22:57.297 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T07:22:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.413+0000 7f10eb143700 10 monclient: get_monmap_and_config 2024-09-16T07:22:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.413+0000 7f10eb143700 10 monclient: build_initial_monmap 2024-09-16T07:22:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.413+0000 7f10eb143700 10 monclient: monmap: 2024-09-16T07:22:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:57.419263+0000 2024-09-16T07:22:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:57.419263+0000 2024-09-16T07:22:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:57.420 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 Processor -- start 2024-09-16T07:22:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 -- start start 2024-09-16T07:22:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient: init 2024-09-16T07:22:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10e40a2750 addr v1:172.21.15.143:6789/0 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f10e40a3240 0x7f10e41382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient(hunting): picked mon.noname-a con 0x7f10e40a3240 addr v2:172.21.15.37:3300/0 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f10e4138830 0x7f10e413ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient(hunting): picked mon.noname-c con 0x7f10e4138830 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e4099be0 con 0x7f10e40a2750 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e409de40 con 0x7f10e40a3240 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e409d530 con 0x7f10e4138830 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient(hunting): authenticate will time out at 1097.905093s 2024-09-16T07:22:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f10e40a3240 0x7f10e41382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f10e40a3240 0x7f10e41382f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44126/0 (socket says 172.21.15.37:44126) 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 1 -- 172.21.15.37:0/2813053713 learned_addr learned my addr 172.21.15.37:0/2813053713 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 10 monclient(hunting): get_auth_request con 0x7f10e40a3240 auth_method 0 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10ea942700 1 --2- 172.21.15.37:0/2813053713 >> v2:172.21.15.143:3300/0 conn(0x7f10e4138830 0x7f10e413ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10ea942700 10 monclient(hunting): get_auth_request con 0x7f10e4138830 auth_method 0 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10ea942700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10ea942700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10ea942700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 10 monclient(hunting): handle_auth_done global_id 34794 payload 274 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 1 -- 172.21.15.37:0/2813053713 >> v2:172.21.15.143:3300/0 conn(0x7f10e4138830 msgr2=0x7f10e413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 1 --2- 172.21.15.37:0/2813053713 >> v2:172.21.15.143:3300/0 conn(0x7f10e4138830 0x7f10e413ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 1 -- 172.21.15.37:0/2813053713 >> v1:172.21.15.143:6789/0 conn(0x7f10e40a2750 legacy=0x7f10e40a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 10 monclient: _finish_hunting 0 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 1 monclient: found mon.noname-a 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 1 -- 172.21.15.37:0/2813053713 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e409a020 con 0x7f10e40a3240 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e9940700 1 --2- 172.21.15.37:0/2813053713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a3240 0x7f10e41382f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f10e40a36b0 tx=0x7f10d4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ce95d7fc668502 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 1 -- 172.21.15.37:0/2813053713 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d400b8a0 con 0x7f10e40a3240 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 10 monclient: dump: 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 10 monclient: _finish_auth 0 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 10 monclient: _check_auth_tickets 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 5 monclient: authenticate success, global_id 34794 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 1 -- 172.21.15.37:0/2813053713 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d400ba00 con 0x7f10e40a3240 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 1 -- 172.21.15.37:0/2813053713 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d400bd90 con 0x7f10e40a3240 2024-09-16T07:22:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 4 set_mon_vals no callback set 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10e913f700 10 monclient: dump: 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.433 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient: shutdown 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 -- 172.21.15.37:0/2813053713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a3240 msgr2=0x7f10e41382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 --2- 172.21.15.37:0/2813053713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a3240 0x7f10e41382f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f10e40a36b0 tx=0x7f10d4009bf0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 -- 172.21.15.37:0/2813053713 shutdown_connections 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 --2- 172.21.15.37:0/2813053713 >> v2:172.21.15.143:3300/0 conn(0x7f10e4138830 0x7f10e413ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 --2- 172.21.15.37:0/2813053713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a3240 0x7f10e41382f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 -- 172.21.15.37:0/2813053713 >> 172.21.15.37:0/2813053713 conn(0x7f10e40a3ea0 msgr2=0x7f10e40a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 -- 172.21.15.37:0/2813053713 shutdown_connections 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 1 -- 172.21.15.37:0/2813053713 wait complete. 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient: build_initial_monmap 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.417+0000 7f10eb143700 10 monclient: monmap: 2024-09-16T07:22:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 Processor -- start 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 -- start start 2024-09-16T07:22:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient: init 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a2750 0x7f10e41c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient(hunting): picked mon.noname-a con 0x7f10e40a2750 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e40a3240 0x7f10e4137680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient(hunting): picked mon.noname-b con 0x7f10e40a3240 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10e4138830 0x7f10e41200b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient(hunting): picked mon.noname-c con 0x7f10e4138830 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e4123870 con 0x7f10e40a2750 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e41c8e20 con 0x7f10e40a3240 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e406d260 con 0x7f10e4138830 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient(hunting): authenticate will time out at 1097.908678s 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a2750 0x7f10e41c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea942700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10e4138830 0x7f10e41200b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a2750 0x7f10e41c7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44142/0 (socket says 172.21.15.37:44142) 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 1 -- 172.21.15.37:0/2353718675 learned_addr learned my addr 172.21.15.37:0/2353718675 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 10 monclient(hunting): get_auth_request con 0x7f10e40a2750 auth_method 0 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10e9940700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e40a3240 0x7f10e4137680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10e9940700 10 monclient(hunting): get_auth_request con 0x7f10e40a3240 auth_method 0 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10e9940700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10e9940700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10e9940700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea942700 10 monclient(hunting): get_auth_request con 0x7f10e4138830 auth_method 0 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea942700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea942700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea942700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10e9940700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10e9940700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10e9940700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea942700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea942700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea942700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 10 monclient(hunting): handle_auth_done global_id 34797 payload 824 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 1 -- 172.21.15.37:0/2353718675 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10e4138830 msgr2=0x7f10e41200b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10e4138830 0x7f10e41200b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 1 -- 172.21.15.37:0/2353718675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e40a3240 msgr2=0x7f10e4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e40a3240 0x7f10e4137680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 10 monclient: _finish_hunting 0 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 1 monclient: found mon.noname-a 2024-09-16T07:22:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 1 -- 172.21.15.37:0/2353718675 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d4009570 con 0x7f10e40a2750 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10ea141700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a2750 0x7f10e41c7e40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00cc20 tx=0x7f10dc00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 1 -- 172.21.15.37:0/2353718675 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc003c20 con 0x7f10e40a2750 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 10 monclient: dump: 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.441 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 10 monclient: _finish_auth 0 2024-09-16T07:22:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 10 monclient: _check_auth_tickets 2024-09-16T07:22:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 1 -- 172.21.15.37:0/2353718675 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10dc003d80 con 0x7f10e40a2750 2024-09-16T07:22:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 1 -- 172.21.15.37:0/2353718675 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10dc00aae0 con 0x7f10e40a2750 2024-09-16T07:22:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 10 monclient: dump: 2024-09-16T07:22:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 5 monclient: authenticate success, global_id 34797 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient: _renew_subs 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10e893e700 4 set_mon_vals no callback set 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 -- 172.21.15.37:0/2353718675 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e4122830 con 0x7f10e40a2750 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient: _renew_subs 2024-09-16T07:22:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 -- 172.21.15.37:0/2353718675 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e406fdf0 con 0x7f10e40a2750 2024-09-16T07:22:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10eb143700 1 -- 172.21.15.37:0/2353718675 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e40a2b30 con 0x7f10e40a2750 2024-09-16T07:22:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.421+0000 7f10daffd700 1 -- 172.21.15.37:0/2353718675 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f10dc00ac40 con 0x7f10e40a2750 2024-09-16T07:22:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.425+0000 7f10daffd700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f10c8035600 0x7f10c8037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.425+0000 7f10daffd700 1 -- 172.21.15.37:0/2353718675 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f10dc047bf0 con 0x7f10e40a2750 2024-09-16T07:22:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.425+0000 7f10e9940700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f10c8035600 0x7f10c8037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.425+0000 7f10e9940700 10 monclient: get_auth_request con 0x7f10c8035600 auth_method 0 2024-09-16T07:22:57.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.425+0000 7f10e9940700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f10c8035600 0x7f10c8037ac0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f10d4000c00 tx=0x7f10d4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:57.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.429+0000 7f10daffd700 1 -- 172.21.15.37:0/2353718675 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f10dc0198f0 con 0x7f10e40a2750 2024-09-16T07:22:57.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.429+0000 7f10daffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:57.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.429+0000 7f10daffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:57.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.665+0000 7f10eb143700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T07:22:57.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.665+0000 7f10eb143700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T07:22:57.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.665+0000 7f10eb143700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:57.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.665+0000 7f10eb143700 1 -- 172.21.15.37:0/2353718675 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f10e4070d70 con 0x7f10e40a2750 2024-09-16T07:22:57.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.673+0000 7f10daffd700 1 -- 172.21.15.37:0/2353718675 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f10dc019300 con 0x7f10e40a2750 2024-09-16T07:22:57.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.673+0000 7f10daffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T07:22:57.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.673+0000 7f10daffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:57.681 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.681 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=11,kv=12,logm=65} crc {auth=811294314,config=3541143443,health=425936381,kv=3696321355,logm=2595781365}) 2024-09-16T07:22:57.681 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=1854360818,config=3541143443,health=2157093694,kv=3014237909,logm=670191091}) 2024-09-16T07:22:57.681 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.681 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=11,kv=12,logm=65} crc {auth=811294314,config=3541143443,health=425936381,kv=3696321355,logm=2595781365}) 2024-09-16T07:22:57.681 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=11,kv=11,logm=65} crc {auth=1336119457,config=3541143443,health=1886805112,kv=1136229706,logm=3273564342}) 2024-09-16T07:22:57.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 -- 172.21.15.37:0/2353718675 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f10c8035600 msgr2=0x7f10c8037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f10c8035600 0x7f10c8037ac0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f10d4000c00 tx=0x7f10d4008030 comp rx=0 tx=0).stop 2024-09-16T07:22:57.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 10 monclient: shutdown 2024-09-16T07:22:57.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:57.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 -- 172.21.15.37:0/2353718675 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a2750 msgr2=0x7f10e41c7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:57.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a2750 0x7f10e41c7e40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00cc20 tx=0x7f10dc00cf30 comp rx=0 tx=0).stop 2024-09-16T07:22:57.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 -- 172.21.15.37:0/2353718675 shutdown_connections 2024-09-16T07:22:57.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f10c8035600 0x7f10c8037ac0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10e4138830 0x7f10e41200b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10e40a3240 0x7f10e4137680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 --2- 172.21.15.37:0/2353718675 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10e40a2750 0x7f10e41c7e40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:57.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 -- 172.21.15.37:0/2353718675 >> 172.21.15.37:0/2353718675 conn(0x7f10e40a3ea0 msgr2=0x7f10e406f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:57.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 -- 172.21.15.37:0/2353718675 shutdown_connections 2024-09-16T07:22:57.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:57.677+0000 7f10eb143700 1 -- 172.21.15.37:0/2353718675 wait complete. 2024-09-16T07:22:57.686 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.686 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2266318102}) 2024-09-16T07:22:57.686 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058111018}) 2024-09-16T07:22:57.687 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.687 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2266318102}) 2024-09-16T07:22:57.687 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1362527726}) 2024-09-16T07:22:57.687 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.687 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1819842336}) 2024-09-16T07:22:57.687 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607322051}) 2024-09-16T07:22:57.687 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.687 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1819842336}) 2024-09-16T07:22:57.687 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3076227357}) 2024-09-16T07:22:57.688 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.688 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1336679131}) 2024-09-16T07:22:57.688 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1459084307}) 2024-09-16T07:22:57.688 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.688 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1336679131}) 2024-09-16T07:22:57.688 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.682+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3883802807}) 2024-09-16T07:22:57.688 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.688 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327504461}) 2024-09-16T07:22:57.688 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600217723}) 2024-09-16T07:22:57.688 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327504461}) 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=713398447}) 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153630876}) 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15239090}) 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153630876}) 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1761234885}) 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1119030324}) 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670831987}) 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.689 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1119030324}) 2024-09-16T07:22:57.690 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4018117131}) 2024-09-16T07:22:57.692 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.692 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2807658475}) 2024-09-16T07:22:57.692 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244398259}) 2024-09-16T07:22:57.692 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.692 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2807658475}) 2024-09-16T07:22:57.692 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.686+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499323558}) 2024-09-16T07:22:57.692 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3707521010}) 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1360575135}) 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3707521010}) 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1870444370}) 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=66,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13} crc {logm=2640769577,mdsmap=1531151339,mgr=2249125188,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=3604937923,mon_config_key=2556715947}) 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13,monmap=3,osd_metadata=1} crc {logm=1769962252,mdsmap=3140514009,mgr=49779396,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=3946782844,mon_config_key=3043435045,monmap=1839276543,osd_metadata=3985140351}) 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=66,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13} crc {logm=2640769577,mdsmap=1531151339,mgr=2249125188,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=3604937923,mon_config_key=2556715947}) 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13} crc {logm=2433116009,mdsmap=1531151339,mgr=49779396,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=1607731953,mon_config_key=3199233330}) 2024-09-16T07:22:57.693 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=1,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {mon_config_key=2286397662,monmap=38151330,osd_metadata=438397694,osd_pg_creating=4178827905,osd_snap=1614000649,osdmap=701355995}) 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=88} crc {mon_config_key=2286397662,monmap=1839276543,osd_metadata=598972236,osd_pg_creating=4178827905,osd_snap=3829634508,osdmap=509569946}) 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=1,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {mon_config_key=2286397662,monmap=38151330,osd_metadata=438397694,osd_pg_creating=4178827905,osd_snap=1614000649,osdmap=701355995}) 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.690+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=1,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=87} crc {mon_config_key=2286397662,monmap=1839276543,osd_metadata=598972236,osd_pg_creating=4178827905,osd_snap=1614000649,osdmap=2230598807}) 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3087803734}) 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4141663626}) 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3087803734}) 2024-09-16T07:22:57.694 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=510938153}) 2024-09-16T07:22:57.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=428014814}) 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1044617388}) 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=428014814}) 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3410240983}) 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=1164305742}) 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=3930201422}) 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=1164305742}) 2024-09-16T07:22:57.697 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:22:57.694+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=52} crc {osdmap=1983781576}) 2024-09-16T07:22:57.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27327]: starting. 2024-09-16T07:22:57.809 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27327]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27326 2024-09-16T07:22:57.809 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27327]: created object 0... 2024-09-16T07:22:57.809 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27327]: created object 25... 2024-09-16T07:22:57.809 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27327]: created object 49... 2024-09-16T07:22:57.809 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27327]: finishing. 2024-09-16T07:22:57.809 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27327]: shutting down. 2024-09-16T07:22:58.689 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27328]: starting. 2024-09-16T07:22:58.689 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27328]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27326 2024-09-16T07:22:58.689 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27328]: shutting down. 2024-09-16T07:22:58.697 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T07:22:58.697 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.801+0000 7f33d885e700 10 monclient: get_monmap_and_config 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.801+0000 7f33d885e700 10 monclient: build_initial_monmap 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.801+0000 7f33d885e700 10 monclient: monmap: 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:58.806447+0000 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:58.806447+0000 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:58.806 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:58.807 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:58.807 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:58.807 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:58.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.801+0000 7f33d885e700 1 Processor -- start 2024-09-16T07:22:58.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.801+0000 7f33d885e700 1 -- start start 2024-09-16T07:22:58.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.801+0000 7f33d885e700 10 monclient: init 2024-09-16T07:22:58.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:58.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T07:22:58.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d406d3c0 0x7f33d406d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:58.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient(hunting): picked mon.noname-b con 0x7f33d406d3c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:58.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f33d40a10a0 0x7f33d406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient(hunting): picked mon.noname-c con 0x7f33d40a10a0 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33d40a06b0 addr v1:172.21.15.143:6789/0 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d406e220 con 0x7f33d40a06b0 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d406f050 con 0x7f33d40a10a0 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d406f330 con 0x7f33d406d3c0 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient(hunting): authenticate will time out at 1099.292612s 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d406d3c0 0x7f33d406d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d406d3c0 0x7f33d406d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50602/0 (socket says 172.21.15.37:50602) 2024-09-16T07:22:58.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 1 -- 172.21.15.37:0/2761035643 learned_addr learned my addr 172.21.15.37:0/2761035643 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 10 monclient(hunting): get_auth_request con 0x7f33d406d3c0 auth_method 0 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33ca59c700 1 --2- 172.21.15.37:0/2761035643 >> v2:172.21.15.143:3300/0 conn(0x7f33d40a10a0 0x7f33d406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33ca59c700 10 monclient(hunting): get_auth_request con 0x7f33d40a10a0 auth_method 0 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33ca59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33ca59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33ca59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:58.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33ca59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33ca59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33ca59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 10 monclient(hunting): handle_auth_done global_id 44123 payload 274 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 1 -- 172.21.15.37:0/2761035643 >> v2:172.21.15.143:3300/0 conn(0x7f33d40a10a0 msgr2=0x7f33d406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 1 --2- 172.21.15.37:0/2761035643 >> v2:172.21.15.143:3300/0 conn(0x7f33d40a10a0 0x7f33d406dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 1 -- 172.21.15.37:0/2761035643 >> v1:172.21.15.143:6789/0 conn(0x7f33d40a06b0 legacy=0x7f33d41321f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 10 monclient: _finish_hunting 0 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 1 monclient: found mon.noname-b 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 1 -- 172.21.15.37:0/2761035643 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d409ee80 con 0x7f33d406d3c0 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d2d9d700 1 --2- 172.21.15.37:0/2761035643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d406d3c0 0x7f33d406d7a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009910 tx=0x7f33bc009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=590e38c9852ce8b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:58.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 1 -- 172.21.15.37:0/2761035643 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc011830 con 0x7f33d406d3c0 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 10 monclient: dump: 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 10 monclient: _finish_auth 0 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 10 monclient: _check_auth_tickets 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 5 monclient: authenticate success, global_id 44123 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 1 -- 172.21.15.37:0/2761035643 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc011990 con 0x7f33d406d3c0 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 4 set_mon_vals no callback set 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 1 -- 172.21.15.37:0/2761035643 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc011d40 con 0x7f33d406d3c0 2024-09-16T07:22:58.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d259c700 10 monclient: dump: 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 10 monclient: shutdown 2024-09-16T07:22:58.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:58.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 -- 172.21.15.37:0/2761035643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d406d3c0 msgr2=0x7f33d406d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:58.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 --2- 172.21.15.37:0/2761035643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d406d3c0 0x7f33d406d7a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009910 tx=0x7f33bc009c20 comp rx=0 tx=0).stop 2024-09-16T07:22:58.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 -- 172.21.15.37:0/2761035643 shutdown_connections 2024-09-16T07:22:58.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 --2- 172.21.15.37:0/2761035643 >> v2:172.21.15.143:3300/0 conn(0x7f33d40a10a0 0x7f33d406dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:58.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 --2- 172.21.15.37:0/2761035643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d406d3c0 0x7f33d406d7a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:58.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 -- 172.21.15.37:0/2761035643 >> 172.21.15.37:0/2761035643 conn(0x7f33d40a1530 msgr2=0x7f33d40a1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:58.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 -- 172.21.15.37:0/2761035643 shutdown_connections 2024-09-16T07:22:58.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.805+0000 7f33d885e700 1 -- 172.21.15.37:0/2761035643 wait complete. 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient: build_initial_monmap 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient: monmap: 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:58.824 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 1 Processor -- start 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 1 -- start start 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient: init 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33d406d3c0 0x7f33d406a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient(hunting): picked mon.noname-b con 0x7f33d406d3c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d40a06b0 0x7f33d4131710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient(hunting): picked mon.noname-a con 0x7f33d40a06b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d40a10a0 0x7f33d4131c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient(hunting): picked mon.noname-c con 0x7f33d40a10a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:58.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d4120170 con 0x7f33d40a06b0 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40726f0 con 0x7f33d406d3c0 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d406b2d0 con 0x7f33d40a10a0 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d885e700 10 monclient(hunting): authenticate will time out at 1099.299322s 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33ca59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d40a06b0 0x7f33d4131710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33ca59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d40a06b0 0x7f33d4131710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44152/0 (socket says 172.21.15.37:44152) 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33ca59c700 1 -- 172.21.15.37:0/2908057446 learned_addr learned my addr 172.21.15.37:0/2908057446 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33ca59c700 10 monclient(hunting): get_auth_request con 0x7f33d40a06b0 auth_method 0 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33ca59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33ca59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:58.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33ca59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d2d9d700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33d406d3c0 0x7f33d406a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d359e700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d40a10a0 0x7f33d4131c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d2d9d700 10 monclient(hunting): get_auth_request con 0x7f33d406d3c0 auth_method 0 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33ca59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33ca59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33ca59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d359e700 10 monclient(hunting): get_auth_request con 0x7f33d40a10a0 auth_method 0 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d359e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:58.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.809+0000 7f33d2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 10 monclient(hunting): handle_auth_done global_id 34818 payload 824 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 1 -- 172.21.15.37:0/2908057446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d40a10a0 msgr2=0x7f33d4131c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d40a10a0 0x7f33d4131c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 1 -- 172.21.15.37:0/2908057446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33d406d3c0 msgr2=0x7f33d406a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33d406d3c0 0x7f33d406a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 10 monclient: _finish_hunting 0 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 1 monclient: found mon.noname-a 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:58.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 1 -- 172.21.15.37:0/2908057446 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc009570 con 0x7f33d40a06b0 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33ca59c700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d40a06b0 0x7f33d4131710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f33c400e9a0 tx=0x7f33c400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 1 -- 172.21.15.37:0/2908057446 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4009720 con 0x7f33d40a06b0 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 10 monclient: dump: 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:58.829 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 10 monclient: _finish_auth 0 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 10 monclient: _check_auth_tickets 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 5 monclient: authenticate success, global_id 34818 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 10 monclient: _renew_subs 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 1 -- 172.21.15.37:0/2908057446 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c400c850 con 0x7f33d40a06b0 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 1 -- 172.21.15.37:0/2908057446 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d409da60 con 0x7f33d40a06b0 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 1 -- 172.21.15.37:0/2908057446 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c400cc00 con 0x7f33d40a06b0 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:58.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 10 monclient: dump: 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d159a700 4 set_mon_vals no callback set 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 10 monclient: _renew_subs 2024-09-16T07:22:58.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 1 -- 172.21.15.37:0/2908057446 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d411f840 con 0x7f33d40a06b0 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d885e700 1 -- 172.21.15.37:0/2908057446 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d4067890 con 0x7f33d40a06b0 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 1 -- 172.21.15.37:0/2908057446 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f33c4013470 con 0x7f33d40a06b0 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33b8035600 0x7f33b8037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33cbfff700 1 -- 172.21.15.37:0/2908057446 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f33c4047a90 con 0x7f33d40a06b0 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d2d9d700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33b8035600 0x7f33b8037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.813+0000 7f33d2d9d700 10 monclient: get_auth_request con 0x7f33b8035600 auth_method 0 2024-09-16T07:22:58.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.817+0000 7f33d2d9d700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33b8035600 0x7f33b8037ac0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009c60 tx=0x7f33bc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:58.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.821+0000 7f33cbfff700 1 -- 172.21.15.37:0/2908057446 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f33c401b3b0 con 0x7f33d40a06b0 2024-09-16T07:22:58.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.821+0000 7f33cbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:58.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:58.821+0000 7f33cbfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.057+0000 7f33d885e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:22:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.057+0000 7f33d885e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.057+0000 7f33d885e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:59.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.057+0000 7f33d885e700 1 -- 172.21.15.37:0/2908057446 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f33d411d7b0 con 0x7f33d40a06b0 2024-09-16T07:22:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.057+0000 7f33cbfff700 1 -- 172.21.15.37:0/2908057446 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f33c4017070 con 0x7f33d40a06b0 2024-09-16T07:22:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.057+0000 7f33cbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:22:59.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.057+0000 7f33cbfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:59.061 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:22:59.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 -- 172.21.15.37:0/2908057446 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33b8035600 msgr2=0x7f33b8037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33b8035600 0x7f33b8037ac0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009c60 tx=0x7f33bc008030 comp rx=0 tx=0).stop 2024-09-16T07:22:59.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 10 monclient: shutdown 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 -- 172.21.15.37:0/2908057446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d40a06b0 msgr2=0x7f33d4131710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d40a06b0 0x7f33d4131710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f33c400e9a0 tx=0x7f33c400ecb0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 -- 172.21.15.37:0/2908057446 shutdown_connections 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33b8035600 0x7f33b8037ac0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d40a10a0 0x7f33d4131c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d40a06b0 0x7f33d4131710 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 --2- 172.21.15.37:0/2908057446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f33d406d3c0 0x7f33d406a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 -- 172.21.15.37:0/2908057446 >> 172.21.15.37:0/2908057446 conn(0x7f33d40a1530 msgr2=0x7f33d40a32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 -- 172.21.15.37:0/2908057446 shutdown_connections 2024-09-16T07:22:59.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.061+0000 7f33d885e700 1 -- 172.21.15.37:0/2908057446 wait complete. 2024-09-16T07:22:59.080 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T07:22:59.081 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T07:22:59.081 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T07:22:59.081 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient: get_monmap_and_config 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient: build_initial_monmap 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient: monmap: 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:59.197976+0000 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:59.197976+0000 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:59.198 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:59.199 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:59.199 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:59.199 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:59.199 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 1 Processor -- start 2024-09-16T07:22:59.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 1 -- start start 2024-09-16T07:22:59.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient: init 2024-09-16T07:22:59.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:59.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-16T07:22:59.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff2d80a0e90 addr v1:172.21.15.143:6789/0 2024-09-16T07:22:59.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff2d80a04d0 0x7ff2d80a1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient(hunting): picked mon.noname-c con 0x7ff2d80a04d0 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:59.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2d80a2d50 addr v1:172.21.15.37:6789/0 2024-09-16T07:22:59.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d80a0900 con 0x7ff2d80a2d50 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2d809f250 con 0x7ff2d80a0e90 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2d80a17b0 con 0x7ff2d80a04d0 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.193+0000 7ff2dd657700 10 monclient(hunting): authenticate will time out at 1099.683938s 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dce56700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff2d80a2d50 0x7ff2d80a3180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47248/0 (socket says 172.21.15.37:47248) 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dce56700 1 -- 172.21.15.37:0/2412881697 learned_addr learned my addr 172.21.15.37:0/2412881697 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 1 --2- 172.21.15.37:0/2412881697 >> v2:172.21.15.143:3300/0 conn(0x7ff2d80a04d0 0x7ff2d80a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 10 monclient(hunting): get_auth_request con 0x7ff2d80a04d0 auth_method 0 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 1 -- 172.21.15.37:0/2412881697 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112489642 0 0) 0x7ff2d80a0900 con 0x7ff2d80a2d50 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient(hunting): _negotiate 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient(hunting): my global_id is 34821 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 1 -- 172.21.15.37:0/2412881697 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c00035d0 con 0x7ff2d80a2d50 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 10 monclient(hunting): handle_auth_done global_id 5734 payload 274 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 1 -- 172.21.15.37:0/2412881697 >> v1:172.21.15.143:6789/0 conn(0x7ff2d80a0e90 legacy=0x7ff2d80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 1 -- 172.21.15.37:0/2412881697 >> v1:172.21.15.37:6789/0 conn(0x7ff2d80a2d50 legacy=0x7ff2d80a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 10 monclient: _finish_hunting 0 2024-09-16T07:22:59.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 1 monclient: found mon.noname-c 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 1 -- 172.21.15.37:0/2412881697 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d809f530 con 0x7ff2d80a04d0 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d77fe700 1 --2- 172.21.15.37:0/2412881697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2d80a04d0 0x7ff2d80a1270 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009dc0 tx=0x7ff2c800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75105ed137446aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 1 -- 172.21.15.37:0/2412881697 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c800e070 con 0x7ff2d80a04d0 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient: dump: 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient: _finish_auth 0 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 5 monclient: authenticate success, global_id 5734 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 1 -- 172.21.15.37:0/2412881697 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c800bae0 con 0x7ff2d80a04d0 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 1 -- 172.21.15.37:0/2412881697 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c800be50 con 0x7ff2d80a04d0 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 4 set_mon_vals no callback set 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2d6ffd700 10 monclient: dump: 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 10 monclient: shutdown 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 1 -- 172.21.15.37:0/2412881697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2d80a04d0 msgr2=0x7ff2d80a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 1 --2- 172.21.15.37:0/2412881697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2d80a04d0 0x7ff2d80a1270 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009dc0 tx=0x7ff2c800b330 comp rx=0 tx=0).stop 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 1 -- 172.21.15.37:0/2412881697 shutdown_connections 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 1 --2- 172.21.15.37:0/2412881697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2d80a04d0 0x7ff2d80a1270 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 1 -- 172.21.15.37:0/2412881697 >> 172.21.15.37:0/2412881697 conn(0x7ff2d80a4740 msgr2=0x7ff2d80a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 1 -- 172.21.15.37:0/2412881697 shutdown_connections 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 1 -- 172.21.15.37:0/2412881697 wait complete. 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 10 monclient: build_initial_monmap 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.197+0000 7ff2dd657700 10 monclient: monmap: 2024-09-16T07:22:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 Processor -- start 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 -- start start 2024-09-16T07:22:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient: init 2024-09-16T07:22:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:22:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2d80a04d0 0x7ff2d81ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient(hunting): picked mon.noname-c con 0x7ff2d80a04d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d80a0e90 0x7ff2d81caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient(hunting): picked mon.noname-a con 0x7ff2d80a0e90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2d80a2d50 0x7ff2d8122a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient(hunting): picked mon.noname-b con 0x7ff2d80a2d50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d8126510 con 0x7ff2d80a0e90 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d8126760 con 0x7ff2d80a2d50 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d8125320 con 0x7ff2d80a04d0 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient(hunting): authenticate will time out at 1099.689505s 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d80a0e90 0x7ff2d81caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d7fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2d80a04d0 0x7ff2d81ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d80a0e90 0x7ff2d81caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44154/0 (socket says 172.21.15.37:44154) 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 1 -- 172.21.15.37:0/1698544213 learned_addr learned my addr 172.21.15.37:0/1698544213 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 10 monclient(hunting): get_auth_request con 0x7ff2d80a0e90 auth_method 0 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d7fff700 10 monclient(hunting): get_auth_request con 0x7ff2d80a04d0 auth_method 0 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dce56700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2d80a2d50 0x7ff2d8122a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dce56700 10 monclient(hunting): get_auth_request con 0x7ff2d80a2d50 auth_method 0 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dce56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dce56700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dce56700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dce56700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dce56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dce56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 10 monclient(hunting): handle_auth_done global_id 34824 payload 824 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 1 -- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2d80a04d0 msgr2=0x7ff2d81ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2d80a04d0 0x7ff2d81ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 1 -- 172.21.15.37:0/1698544213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2d80a2d50 msgr2=0x7ff2d8122a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2d80a2d50 0x7ff2d8122a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 10 monclient: _finish_hunting 0 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 1 monclient: found mon.noname-a 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 1 -- 172.21.15.37:0/1698544213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c8009a70 con 0x7ff2d80a0e90 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d77fe700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d80a0e90 0x7ff2d81caba0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80a46a0 tx=0x7ff2c8004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8027480 con 0x7ff2d80a0e90 2024-09-16T07:22:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 10 monclient: dump: 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 10 monclient: _finish_auth 0 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c8020070 con 0x7ff2d80a0e90 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 5 monclient: authenticate success, global_id 34824 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient: _renew_subs 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 -- 172.21.15.37:0/1698544213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d806be40 con 0x7ff2d80a0e90 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8027820 con 0x7ff2d80a0e90 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d4ff9700 10 monclient: dump: 2024-09-16T07:22:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2d67fc700 4 set_mon_vals no callback set 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient: _renew_subs 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 -- 172.21.15.37:0/1698544213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d8125030 con 0x7ff2d80a0e90 2024-09-16T07:22:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d8122f80 0x7ff2d8123360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.201+0000 7ff2dd657700 1 -- 172.21.15.37:0/1698544213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d81268f0 con 0x7ff2d8122f80 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7ff2c801c400 con 0x7ff2d80a0e90 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d8122f80 0x7ff2d8123360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 10 monclient: get_auth_request con 0x7ff2d8122f80 auth_method 0 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff2c4035600 0x7ff2c4037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 4346+0+0 (secure 0 0 0) 0x7ff2c8022070 con 0x7ff2d80a0e90 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2dce56700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff2c4035600 0x7ff2c4037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2dce56700 10 monclient: get_auth_request con 0x7ff2c4035600 auth_method 0 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_done global_id 34827 payload 824 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 1 -- 172.21.15.37:0/1698544213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2d813e460 con 0x7ff2d8122f80 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d7fff700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d8122f80 0x7ff2d8123360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00f7a0 tx=0x7ff2cc00fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc005320 con 0x7ff2d8122f80 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 10 monclient: dump: 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2dce56700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff2c4035600 0x7ff2c4037ac0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff2d813dbe0 tx=0x7ff2d0007d60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ff2cc005480 con 0x7ff2d8122f80 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d8122f80 msgr2=0x7ff2d8123360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d8122f80 0x7ff2d8123360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00f7a0 tx=0x7ff2cc00fb60 comp rx=0 tx=0).stop 2024-09-16T07:22:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d8122f80 msgr2=0x7ff2d8123360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:59.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.205+0000 7ff2d4ff9700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d8122f80 0x7ff2d8123360 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00f7a0 tx=0x7ff2cc00fb60 comp rx=0 tx=0).stop 2024-09-16T07:22:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2dd657700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2dd657700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d81238a0 0x7ff2d811f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2dd657700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2dd657700 1 -- 172.21.15.37:0/1698544213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d8067400 con 0x7ff2d81238a0 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2d7fff700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d81238a0 0x7ff2d811f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2d7fff700 10 monclient: get_auth_request con 0x7ff2d81238a0 auth_method 0 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.213+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d7fff700 10 monclient(hunting): handle_auth_done global_id 34830 payload 824 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d7fff700 1 -- 172.21.15.37:0/1698544213 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff2d809dea0 con 0x7ff2d81238a0 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d7fff700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d81238a0 0x7ff2d811f2a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00f7a0 tx=0x7ff2cc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc023070 con 0x7ff2d81238a0 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 10 monclient: dump: 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4619 (secure 0 0 0) 0x7ff2cc007600 con 0x7ff2d81238a0 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d81238a0 msgr2=0x7ff2d811f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d81238a0 0x7ff2d811f2a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00f7a0 tx=0x7ff2cc007f90 comp rx=0 tx=0).stop 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 1 -- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d81238a0 msgr2=0x7ff2d811f2a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2d4ff9700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d81238a0 0x7ff2d811f2a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.230 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 22, 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:22:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:22:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:22:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:59.243 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:59.243 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:22:59.243 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.243 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.243 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.243 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:22:59.243 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:22:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dd657700 1 -- 172.21.15.37:0/1698544213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff2c4035600 msgr2=0x7ff2c4037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dd657700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff2c4035600 0x7ff2c4037ac0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff2d813dbe0 tx=0x7ff2d0007d60 comp rx=0 tx=0).stop 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dd657700 10 monclient: shutdown 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dd657700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dd657700 1 -- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d80a0e90 msgr2=0x7ff2d81caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dd657700 1 --2- 172.21.15.37:0/1698544213 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2d80a0e90 0x7ff2d81caba0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff2d80a46a0 tx=0x7ff2c8004af0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dce56700 1 -- 172.21.15.37:0/1698544213 reap_dead start 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dce56700 1 -- 172.21.15.37:0/1698544213 reap_dead start 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dd657700 1 -- 172.21.15.37:0/1698544213 shutdown_connections 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dd657700 1 -- 172.21.15.37:0/1698544213 >> 172.21.15.37:0/1698544213 conn(0x7ff2d80a4740 msgr2=0x7ff2d809c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dd657700 1 -- 172.21.15.37:0/1698544213 shutdown_connections 2024-09-16T07:22:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.217+0000 7ff2dd657700 1 -- 172.21.15.37:0/1698544213 wait complete. 2024-09-16T07:22:59.246 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:22:59.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T07:22:59.246 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T07:22:59.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.357+0000 7fcfd8e9b700 10 monclient: get_monmap_and_config 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.357+0000 7fcfd8e9b700 10 monclient: build_initial_monmap 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.357+0000 7fcfd8e9b700 10 monclient: monmap: 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:59.363139+0000 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:59.363139+0000 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:59.364 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 1 Processor -- start 2024-09-16T07:22:59.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 1 -- start start 2024-09-16T07:22:59.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient: init 2024-09-16T07:22:59.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:59.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T07:22:59.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcfd40a2750 0x7fcfd40a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient(hunting): picked mon.noname-c con 0x7fcfd40a2750 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:59.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcfd40a3070 addr v1:172.21.15.37:6789/0 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd4133730 0x7fcfd413b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient(hunting): picked mon.noname-b con 0x7fcfd4133730 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd4099be0 con 0x7fcfd40a3070 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd409d840 con 0x7fcfd40a2750 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd409de40 con 0x7fcfd4133730 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient(hunting): authenticate will time out at 1099.849012s 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd2ffd700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fcfd40a3070 0x7fcfd4133020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47252/0 (socket says 172.21.15.37:47252) 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd2ffd700 1 -- 172.21.15.37:0/2480602924 learned_addr learned my addr 172.21.15.37:0/2480602924 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:22:59.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd3fff700 1 --2- 172.21.15.37:0/2480602924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd4133730 0x7fcfd413b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd3fff700 10 monclient(hunting): get_auth_request con 0x7fcfd4133730 auth_method 0 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 1 --2- 172.21.15.37:0/2480602924 >> v2:172.21.15.143:3300/0 conn(0x7fcfd40a2750 0x7fcfd40a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 10 monclient(hunting): get_auth_request con 0x7fcfd40a2750 auth_method 0 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 1 -- 172.21.15.37:0/2480602924 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157533712 0 0) 0x7fcfd4099be0 con 0x7fcfd40a3070 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient(hunting): _negotiate 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient(hunting): my global_id is 34833 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 1 -- 172.21.15.37:0/2480602924 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfc00035d0 con 0x7fcfd40a3070 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 10 monclient(hunting): handle_auth_done global_id 5743 payload 274 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 1 -- 172.21.15.37:0/2480602924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd4133730 msgr2=0x7fcfd413b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 1 --2- 172.21.15.37:0/2480602924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd4133730 0x7fcfd413b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 1 -- 172.21.15.37:0/2480602924 >> v1:172.21.15.37:6789/0 conn(0x7fcfd40a3070 legacy=0x7fcfd4133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 10 monclient: _finish_hunting 0 2024-09-16T07:22:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 1 monclient: found mon.noname-c 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 1 -- 172.21.15.37:0/2480602924 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd409d530 con 0x7fcfd40a2750 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd3fff700 1 --2- 172.21.15.37:0/2480602924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd4133730 0x7fcfd413b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd37fe700 1 --2- 172.21.15.37:0/2480602924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd40a2750 0x7fcfd40a2b30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009850 tx=0x7fcfbc009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9865449bdf8b4d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 1 -- 172.21.15.37:0/2480602924 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc011830 con 0x7fcfd40a2750 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient: dump: 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.372 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient: _finish_auth 0 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient: _check_auth_tickets 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 1 -- 172.21.15.37:0/2480602924 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfbc011990 con 0x7fcfd40a2750 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 5 monclient: authenticate success, global_id 5743 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:59.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 1 -- 172.21.15.37:0/2480602924 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc011d40 con 0x7fcfd40a2750 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 4 set_mon_vals no callback set 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd27fc700 10 monclient: dump: 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.374 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 10 monclient: shutdown 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/2480602924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd40a2750 msgr2=0x7fcfd40a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.361+0000 7fcfd8e9b700 1 --2- 172.21.15.37:0/2480602924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd40a2750 0x7fcfd40a2b30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009850 tx=0x7fcfbc009b60 comp rx=0 tx=0).stop 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/2480602924 shutdown_connections 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 --2- 172.21.15.37:0/2480602924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd4133730 0x7fcfd413b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 --2- 172.21.15.37:0/2480602924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd40a2750 0x7fcfd40a2b30 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/2480602924 >> 172.21.15.37:0/2480602924 conn(0x7fcfd40a3ea0 msgr2=0x7fcfd40a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/2480602924 shutdown_connections 2024-09-16T07:22:59.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/2480602924 wait complete. 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient: build_initial_monmap 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient: monmap: 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:59.376 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 Processor -- start 2024-09-16T07:22:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 -- start start 2024-09-16T07:22:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient: init 2024-09-16T07:22:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfd40a2750 0x7fcfd4132200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient(hunting): picked mon.noname-a con 0x7fcfd40a2750 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd40a3070 0x7fcfd4132740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient(hunting): picked mon.noname-b con 0x7fcfd40a3070 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd4133730 0x7fcfd411bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient(hunting): picked mon.noname-c con 0x7fcfd4133730 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd411f320 con 0x7fcfd40a2750 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd411e260 con 0x7fcfd40a3070 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd406cf60 con 0x7fcfd4133730 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd8e9b700 10 monclient(hunting): authenticate will time out at 1099.855406s 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfd40a2750 0x7fcfd4132200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd3fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd4133730 0x7fcfd411bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfd40a2750 0x7fcfd4132200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44202/0 (socket says 172.21.15.37:44202) 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd37fe700 1 -- 172.21.15.37:0/3349425217 learned_addr learned my addr 172.21.15.37:0/3349425217 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd37fe700 10 monclient(hunting): get_auth_request con 0x7fcfd40a2750 auth_method 0 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd3fff700 10 monclient(hunting): get_auth_request con 0x7fcfd4133730 auth_method 0 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd2ffd700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd40a3070 0x7fcfd4132740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd2ffd700 10 monclient(hunting): get_auth_request con 0x7fcfd40a3070 auth_method 0 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.365+0000 7fcfd3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 10 monclient(hunting): handle_auth_done global_id 34836 payload 824 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 1 -- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd4133730 msgr2=0x7fcfd411bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd4133730 0x7fcfd411bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 1 -- 172.21.15.37:0/3349425217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd40a3070 msgr2=0x7fcfd4132740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfd40a3070 0x7fcfd4132740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 10 monclient: _finish_hunting 0 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 1 monclient: found mon.noname-a 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 1 -- 172.21.15.37:0/3349425217 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc0094b0 con 0x7fcfd40a2750 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd37fe700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfd40a2750 0x7fcfd4132200 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc0090b0 tx=0x7fcfbc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc0042c0 con 0x7fcfd40a2750 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: dump: 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.388 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: _finish_auth 0 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: _check_auth_tickets 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 5 monclient: authenticate success, global_id 34836 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfbc01f070 con 0x7fcfd40a2750 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 10 monclient: _renew_subs 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/3349425217 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfbc0094b0 con 0x7fcfd40a2750 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc0036c0 con 0x7fcfd40a2750 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:59.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: dump: 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd1ffb700 4 set_mon_vals no callback set 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 10 monclient: _renew_subs 2024-09-16T07:22:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/3349425217 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd411fbc0 con 0x7fcfd40a2750 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd41c7f90 0x7fcfd411c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/3349425217 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd4067530 con 0x7fcfd41c7f90 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd41c7f90 0x7fcfd411c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient: get_auth_request con 0x7fcfd41c7f90 auth_method 0 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fcfbc021030 con 0x7fcfd40a2750 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_done global_id 44138 payload 824 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 1 -- 172.21.15.37:0/3349425217 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfd40a3480 con 0x7fcfd41c7f90 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcfb4035600 0x7fcfb4037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fcfbc048c10 con 0x7fcfd40a2750 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd2ffd700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd41c7f90 0x7fcfd411c150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcfd411d460 tx=0x7fcfc400f280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd3fff700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcfb4035600 0x7fcfb4037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfd3fff700 10 monclient: get_auth_request con 0x7fcfb4035600 auth_method 0 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400fab0 con 0x7fcfd41c7f90 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: dump: 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fcfc400fc10 con 0x7fcfd41c7f90 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd41c7f90 msgr2=0x7fcfd411c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd41c7f90 0x7fcfd411c150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcfd411d460 tx=0x7fcfc400f280 comp rx=0 tx=0).stop 2024-09-16T07:22:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd41c7f90 msgr2=0x7fcfd411c150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.369+0000 7fcfcbfff700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd41c7f90 0x7fcfd411c150 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcfd411d460 tx=0x7fcfc400f280 comp rx=0 tx=0).stop 2024-09-16T07:22:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.373+0000 7fcfd3fff700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcfb4035600 0x7fcfb4037ac0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009d90 tx=0x7fcfcc009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd8e9b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:22:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd8e9b700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd411c690 0x7fcfd41ba8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd8e9b700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/3349425217 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd411caa0 con 0x7fcfd411c690 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd411c690 0x7fcfd41ba8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 10 monclient: get_auth_request con 0x7fcfd411c690 auth_method 0 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 10 monclient(hunting): handle_auth_done global_id 44141 payload 824 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 1 -- 172.21.15.37:0/3349425217 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcfd411f740 con 0x7fcfd411c690 2024-09-16T07:22:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfd2ffd700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd411c690 0x7fcfd41ba8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4018000 tx=0x7fcfc40014d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4023070 con 0x7fcfd411c690 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 10 monclient: dump: 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.399 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4453 (secure 0 0 0) 0x7fcfc4013800 con 0x7fcfd411c690 2024-09-16T07:22:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:22:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd411c690 msgr2=0x7fcfd41ba8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd411c690 0x7fcfd41ba8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4018000 tx=0x7fcfc40014d0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 1 -- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd411c690 msgr2=0x7fcfd41ba8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.381+0000 7fcfcbfff700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfd411c690 0x7fcfd41ba8c0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4018000 tx=0x7fcfc40014d0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.399 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 22, 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:22:59.400 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:22:59.401 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.402 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:22:59.403 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:59.404 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:22:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:22:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:22:59.411 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:22:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/3349425217 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcfb4035600 msgr2=0x7fcfb4037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fcfb4035600 0x7fcfb4037ac0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009d90 tx=0x7fcfcc009240 comp rx=0 tx=0).stop 2024-09-16T07:22:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 10 monclient: shutdown 2024-09-16T07:22:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfd40a2750 msgr2=0x7fcfd4132200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd3fff700 1 -- 172.21.15.37:0/3349425217 reap_dead start 2024-09-16T07:22:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfd40a2750 0x7fcfd4132200 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc0090b0 tx=0x7fcfbc004770 comp rx=0 tx=0).stop 2024-09-16T07:22:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/3349425217 shutdown_connections 2024-09-16T07:22:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 1 --2- 172.21.15.37:0/3349425217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfd40a2750 0x7fcfd4132200 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/3349425217 >> 172.21.15.37:0/3349425217 conn(0x7fcfd40a3ea0 msgr2=0x7fcfd409f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/3349425217 shutdown_connections 2024-09-16T07:22:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.385+0000 7fcfd8e9b700 1 -- 172.21.15.37:0/3349425217 wait complete. 2024-09-16T07:22:59.414 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:22:59.414 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T07:22:59.414 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T07:22:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient: get_monmap_and_config 2024-09-16T07:22:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient: build_initial_monmap 2024-09-16T07:22:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient: monmap: 2024-09-16T07:22:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:59.536377+0000 2024-09-16T07:22:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:59.536377+0000 2024-09-16T07:22:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.536 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:59.537 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:59.537 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:59.537 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:59.537 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:59.537 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 1 Processor -- start 2024-09-16T07:22:59.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 1 -- start start 2024-09-16T07:22:59.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient: init 2024-09-16T07:22:59.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:59.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd9c06a750 addr v1:172.21.15.37:6789/0 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbd9c06d8f0 0x7fbd9c13a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient(hunting): picked mon.noname-c con 0x7fbd9c06d8f0 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbd9c06b240 0x7fbd9c13cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient(hunting): picked mon.noname-a con 0x7fbd9c06b240 addr v2:172.21.15.37:3300/0 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9c06d460 con 0x7fbd9c06a750 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9c09ec20 con 0x7fbd9c06b240 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9c09ef00 con 0x7fbd9c06d8f0 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient(hunting): authenticate will time out at 1100.022369s 2024-09-16T07:22:59.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda1b88700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fbd9c06a750 0x7fbd9c06ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47258/0 (socket says 172.21.15.37:47258) 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda1b88700 1 -- 172.21.15.37:0/4177504618 learned_addr learned my addr 172.21.15.37:0/4177504618 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 1 --2- 172.21.15.37:0/4177504618 >> v2:172.21.15.37:3300/0 conn(0x7fbd9c06b240 0x7fbd9c13cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 10 monclient(hunting): get_auth_request con 0x7fbd9c06b240 auth_method 0 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda1387700 1 --2- 172.21.15.37:0/4177504618 >> v2:172.21.15.143:3300/0 conn(0x7fbd9c06d8f0 0x7fbd9c13a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda1387700 10 monclient(hunting): get_auth_request con 0x7fbd9c06d8f0 auth_method 0 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda1387700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda1387700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda1387700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 10 monclient(hunting): handle_auth_done global_id 34839 payload 274 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 1 -- 172.21.15.37:0/4177504618 >> v2:172.21.15.143:3300/0 conn(0x7fbd9c06d8f0 msgr2=0x7fbd9c13a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 1 --2- 172.21.15.37:0/4177504618 >> v2:172.21.15.143:3300/0 conn(0x7fbd9c06d8f0 0x7fbd9c13a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 1 -- 172.21.15.37:0/4177504618 >> v1:172.21.15.37:6789/0 conn(0x7fbd9c06a750 legacy=0x7fbd9c06ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 10 monclient: _finish_hunting 0 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 1 monclient: found mon.noname-a 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 1 -- 172.21.15.37:0/4177504618 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807414800 0 0) 0x7fbd9c06d460 con 0x7fbd9c06a750 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 1 -- 172.21.15.37:0/4177504618 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c09e940 con 0x7fbd9c06b240 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:22:59.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2389700 1 --2- 172.21.15.37:0/4177504618 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd9c06b240 0x7fbd9c13cbc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c0a0f00 tx=0x7fbd8c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f0ee16d57d0f75 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 1 -- 172.21.15.37:0/4177504618 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c011830 con 0x7fbd9c06b240 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 10 monclient: dump: 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 10 monclient: _finish_auth 0 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 10 monclient: _check_auth_tickets 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 5 monclient: authenticate success, global_id 34839 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 1 -- 172.21.15.37:0/4177504618 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c011990 con 0x7fbd9c06b240 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda2b8a700 4 set_mon_vals no callback set 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 1 -- 172.21.15.37:0/4177504618 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c011d40 con 0x7fbd9c06b240 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:22:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.533+0000 7fbda0b86700 10 monclient: dump: 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient: shutdown 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 -- 172.21.15.37:0/4177504618 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd9c06b240 msgr2=0x7fbd9c13cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 --2- 172.21.15.37:0/4177504618 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd9c06b240 0x7fbd9c13cbc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c0a0f00 tx=0x7fbd8c009b30 comp rx=0 tx=0).stop 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 -- 172.21.15.37:0/4177504618 shutdown_connections 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 --2- 172.21.15.37:0/4177504618 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd9c06b240 0x7fbd9c13cbc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 --2- 172.21.15.37:0/4177504618 >> v2:172.21.15.143:3300/0 conn(0x7fbd9c06d8f0 0x7fbd9c13a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 -- 172.21.15.37:0/4177504618 >> 172.21.15.37:0/4177504618 conn(0x7fbd9c0a1530 msgr2=0x7fbd9c0a1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 -- 172.21.15.37:0/4177504618 shutdown_connections 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 -- 172.21.15.37:0/4177504618 wait complete. 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient: build_initial_monmap 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient: monmap: 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:59.549 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:59.549 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:59.549 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:59.549 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:59.549 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 Processor -- start 2024-09-16T07:22:59.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 -- start start 2024-09-16T07:22:59.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient: init 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd9c06a750 0x7fbd9c134df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient(hunting): picked mon.noname-c con 0x7fbd9c06a750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd9c06b240 0x7fbd9c135330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient(hunting): picked mon.noname-a con 0x7fbd9c06b240 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c06d8f0 0x7fbd9c11bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient(hunting): picked mon.noname-b con 0x7fbd9c06d8f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c11fa30 con 0x7fbd9c06b240 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c11fc80 con 0x7fbd9c06d8f0 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c072680 con 0x7fbd9c06a750 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda2b8a700 10 monclient(hunting): authenticate will time out at 1100.027627s 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1b88700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd9c06a750 0x7fbd9c134df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1b88700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd9c06a750 0x7fbd9c134df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50648/0 (socket says 172.21.15.37:50648) 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1b88700 1 -- 172.21.15.37:0/462208146 learned_addr learned my addr 172.21.15.37:0/462208146 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1b88700 10 monclient(hunting): get_auth_request con 0x7fbd9c06a750 auth_method 0 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1b88700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1b88700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1b88700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1387700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd9c06b240 0x7fbd9c135330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1b88700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1b88700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1b88700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1387700 10 monclient(hunting): get_auth_request con 0x7fbd9c06b240 auth_method 0 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1387700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1387700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.537+0000 7fbda1387700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2389700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c06d8f0 0x7fbd9c11bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 10 monclient(hunting): handle_auth_done global_id 44144 payload 824 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 1 -- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c06d8f0 msgr2=0x7fbd9c11bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c06d8f0 0x7fbd9c11bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 1 -- 172.21.15.37:0/462208146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd9c06b240 msgr2=0x7fbd9c135330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd9c06b240 0x7fbd9c135330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 10 monclient: _finish_hunting 0 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 1 monclient: found mon.noname-c 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 1 -- 172.21.15.37:0/462208146 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8c0094b0 con 0x7fbd9c06a750 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2389700 10 monclient: get_auth_request con 0x7fbd9c06d8f0 auth_method 0 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2389700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c06d8f0 0x7fbd9c11bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1b88700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd9c06a750 0x7fbd9c134df0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbd94001fe0 tx=0x7fbd9400a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9400ad00 con 0x7fbd9c06a750 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 10 monclient: dump: 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 10 monclient: _finish_auth 0 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:22:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9400ae80 con 0x7fbd9c06a750 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd94007310 con 0x7fbd9c06a750 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 10 monclient: dump: 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 5 monclient: authenticate success, global_id 44144 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 10 monclient: _renew_subs 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9bfff700 4 set_mon_vals no callback set 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 1 -- 172.21.15.37:0/462208146 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c06e210 con 0x7fbd9c06a750 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 10 monclient: _renew_subs 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 1 -- 172.21.15.37:0/462208146 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c11e710 con 0x7fbd9c06a750 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fbd94007470 con 0x7fbd9c06a750 2024-09-16T07:22:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11c9e0 0x7fbd9c02b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2b8a700 1 -- 172.21.15.37:0/462208146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c11fe10 con 0x7fbd9c11c9e0 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbd88035600 0x7fbd88037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fbd94047510 con 0x7fbd9c06a750 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2389700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbd88035600 0x7fbd88037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11c9e0 0x7fbd9c02b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient: get_auth_request con 0x7fbd9c11c9e0 auth_method 0 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2389700 10 monclient: get_auth_request con 0x7fbd88035600 auth_method 0 2024-09-16T07:22:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 10 monclient(hunting): handle_auth_done global_id 5752 payload 824 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda1387700 1 -- 172.21.15.37:0/462208146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd9c13d100 con 0x7fbd9c11c9e0 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.541+0000 7fbda2389700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbd88035600 0x7fbd88037ac0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c11f150 tx=0x7fbd8c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbda1387700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11c9e0 0x7fbd9c02b6a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c06f6c0 tx=0x7fbd90011b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90004440 con 0x7fbd9c11c9e0 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 10 monclient: dump: 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7fbd900045a0 con 0x7fbd9c11c9e0 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11c9e0 msgr2=0x7fbd9c02b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11c9e0 0x7fbd9c02b6a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c06f6c0 tx=0x7fbd90011b90 comp rx=0 tx=0).stop 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11c9e0 msgr2=0x7fbd9c02b6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.545+0000 7fbd9a7fc700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11c9e0 0x7fbd9c02b6a0 secure :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c06f6c0 tx=0x7fbd90011b90 comp rx=0 tx=0).stop 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda2b8a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda2b8a700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11cdc0 0x7fbd9c02bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda2b8a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda2b8a700 1 -- 172.21.15.37:0/462208146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c067490 con 0x7fbd9c11cdc0 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda1387700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11cdc0 0x7fbd9c02bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda1387700 10 monclient: get_auth_request con 0x7fbd9c11cdc0 auth_method 0 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda1387700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda1387700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda1387700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda1387700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda1387700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda1387700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda1387700 10 monclient(hunting): handle_auth_done global_id 5755 payload 824 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.553+0000 7fbda1387700 1 -- 172.21.15.37:0/462208146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbd9c067890 con 0x7fbd9c11cdc0 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda1387700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11cdc0 0x7fbd9c02bbe0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbd90012010 tx=0x7fbd90004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90025070 con 0x7fbd9c11cdc0 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 10 monclient: dump: 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4787 (secure 0 0 0) 0x7fbd900200d0 con 0x7fbd9c11cdc0 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11cdc0 msgr2=0x7fbd9c02bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11cdc0 0x7fbd9c02bbe0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbd90012010 tx=0x7fbd90004770 comp rx=0 tx=0).stop 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 1 -- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11cdc0 msgr2=0x7fbd9c02bbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:22:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbd9a7fc700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c11cdc0 0x7fbd9c02bbe0 secure :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbd90012010 tx=0x7fbd90004770 comp rx=0 tx=0).stop 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 22, 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-09-16T07:22:59.569 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:59.570 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:59.571 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:22:59.572 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:22:59.573 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:59.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:22:59.575 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:59.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:22:59.578 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.579 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-09-16T07:22:59.580 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 8 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:22:59.581 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:22:59.582 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:22:59.582 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:22:59.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:22:59.582 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:22:59.582 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:22:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 1 -- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbd88035600 msgr2=0x7fbd88037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbd88035600 0x7fbd88037ac0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c11f150 tx=0x7fbd8c008030 comp rx=0 tx=0).stop 2024-09-16T07:22:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 10 monclient: shutdown 2024-09-16T07:22:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 1 -- 172.21.15.37:0/462208146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd9c06a750 msgr2=0x7fbd9c134df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd9c06a750 0x7fbd9c134df0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbd94001fe0 tx=0x7fbd9400a340 comp rx=0 tx=0).stop 2024-09-16T07:22:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2389700 1 -- 172.21.15.37:0/462208146 reap_dead start 2024-09-16T07:22:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 1 -- 172.21.15.37:0/462208146 shutdown_connections 2024-09-16T07:22:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 1 --2- 172.21.15.37:0/462208146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd9c06a750 0x7fbd9c134df0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 1 -- 172.21.15.37:0/462208146 >> 172.21.15.37:0/462208146 conn(0x7fbd9c0a1530 msgr2=0x7fbd9c06f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 1 -- 172.21.15.37:0/462208146 shutdown_connections 2024-09-16T07:22:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.557+0000 7fbda2b8a700 1 -- 172.21.15.37:0/462208146 wait complete. 2024-09-16T07:22:59.584 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:22:59.585 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T07:22:59.585 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T07:22:59.585 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T07:22:59.585 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T07:22:59.585 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T07:22:59.585 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:22:59.625 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T07:22:59.625 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T07:22:59.625 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:22:59.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient: get_monmap_and_config 2024-09-16T07:22:59.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient: build_initial_monmap 2024-09-16T07:22:59.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient: monmap: 2024-09-16T07:22:59.745 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:59.745 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.745 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:22:59.745151+0000 2024-09-16T07:22:59.745 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:22:59.745151+0000 2024-09-16T07:22:59.745 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:59.746 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.746 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:22:59.746 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:22:59.746 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:22:59.746 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:22:59.746 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:22:59.746 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 1 Processor -- start 2024-09-16T07:22:59.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 1 -- start start 2024-09-16T07:22:59.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient: init 2024-09-16T07:22:59.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:59.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T07:22:59.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34500a2750 addr v1:172.21.15.37:6789/0 2024-09-16T07:22:59.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3450071880 addr v1:172.21.15.143:6789/0 2024-09-16T07:22:59.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f34500a31e0 0x7f345013a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f34500a31e0 addr v2:172.21.15.143:3300/0 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34500a2b30 con 0x7f34500a2750 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345009c3e0 con 0x7f3450071880 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345009d0e0 con 0x7f34500a31e0 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344f59e700 10 monclient(hunting): authenticate will time out at 1100.231301s 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344e59c700 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f34500a2750 legacy=0x7f34500999e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344ed9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f34500a31e0 0x7f345013a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f3447fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f3450071880 0x7f345009a0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.37:33466/0 (socket says 172.21.15.37:33466) 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f3447fff700 1 -- 172.21.15.37:0/3775565419 learned_addr learned my addr 172.21.15.37:0/3775565419 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344ed9d700 10 monclient(hunting): get_auth_request con 0x7f34500a31e0 auth_method 0 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.741+0000 7f344ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344ed9d700 10 monclient(hunting): handle_auth_done global_id 5758 payload 274 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344ed9d700 1 -- 172.21.15.37:0/3775565419 >> v1:172.21.15.143:6789/0 conn(0x7f3450071880 legacy=0x7f345009a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344ed9d700 1 -- 172.21.15.37:0/3775565419 >> v1:172.21.15.37:6789/0 conn(0x7f34500a2750 legacy=0x7f34500999e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344ed9d700 10 monclient: _finish_hunting 0 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 1 -- 172.21.15.37:0/3775565419 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672988140 0 0) 0x7f345009c3e0 con 0x7f3450071880 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344ed9d700 1 monclient: found mon.noname-c 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344ed9d700 1 -- 172.21.15.37:0/3775565419 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345009a800 con 0x7f34500a31e0 2024-09-16T07:22:59.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344ed9d700 1 --2- 172.21.15.37:0/3775565419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34500a31e0 0x7f345013a060 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f345009d220 tx=0x7f344800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3017f2b4281ef8a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 1 -- 172.21.15.37:0/3775565419 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448016070 con 0x7f34500a31e0 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 10 monclient: dump: 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.753 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 10 monclient: _finish_auth 0 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 10 monclient: _check_auth_tickets 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 5 monclient: authenticate success, global_id 5758 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 1 -- 172.21.15.37:0/3775565419 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344800bb60 con 0x7f34500a31e0 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 1 -- 172.21.15.37:0/3775565419 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344800e070 con 0x7f34500a31e0 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:22:59.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344dd9b700 10 monclient: dump: 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 10 monclient: get_monmap_and_config success 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 4 set_mon_vals no callback set 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 10 monclient: shutdown 2024-09-16T07:22:59.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 1 -- 172.21.15.37:0/3775565419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34500a31e0 msgr2=0x7f345013a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 1 --2- 172.21.15.37:0/3775565419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34500a31e0 0x7f345013a060 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f345009d220 tx=0x7f344800b330 comp rx=0 tx=0).stop 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 1 -- 172.21.15.37:0/3775565419 shutdown_connections 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 1 --2- 172.21.15.37:0/3775565419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34500a31e0 0x7f345013a060 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 1 -- 172.21.15.37:0/3775565419 >> 172.21.15.37:0/3775565419 conn(0x7f34500a3ea0 msgr2=0x7f34500a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 1 -- 172.21.15.37:0/3775565419 shutdown_connections 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 1 -- 172.21.15.37:0/3775565419 wait complete. 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 10 monclient: build_initial_monmap 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.745+0000 7f344f59e700 10 monclient: monmap: 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:22:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:22:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:22:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:22:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:22:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:22:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:22:59.757 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 1 Processor -- start 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 1 -- start start 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient: init 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3450071880 0x7f34501c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f3450071880 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34500a2750 0x7f34501c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f34500a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34500a31e0 0x7f3450120c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f34500a31e0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:22:59.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344e59c700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3450071880 msgr2=0x7f34501c86b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345013c8a0 con 0x7f3450071880 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3450124850 con 0x7f34500a31e0 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3450124aa0 con 0x7f34500a2750 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient(hunting): authenticate will time out at 1100.238094s 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3450071880 0x7f34501c86b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34500a2750 0x7f34501c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34500a2750 0x7f34501c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50686/0 (socket says 172.21.15.37:50686) 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 1 -- 172.21.15.37:0/4014719262 learned_addr learned my addr 172.21.15.37:0/4014719262 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:22:59.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 10 monclient(hunting): get_auth_request con 0x7f34500a2750 auth_method 0 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344ed9d700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34500a31e0 0x7f3450120c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344ed9d700 10 monclient(hunting): get_auth_request con 0x7f34500a31e0 auth_method 0 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 10 monclient(hunting): handle_auth_done global_id 44147 payload 824 2024-09-16T07:22:59.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 1 -- 172.21.15.37:0/4014719262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34500a31e0 msgr2=0x7f3450120c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34500a31e0 0x7f3450120c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 1 -- 172.21.15.37:0/4014719262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3450071880 msgr2=0x7f34501c86b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3450071880 0x7f34501c86b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 10 monclient: _finish_hunting 0 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 1 monclient: found mon.noname-c 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 1 -- 172.21.15.37:0/4014719262 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3448009a70 con 0x7f34500a2750 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3447fff700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34500a2750 0x7f34501c8bf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f343800c520 tx=0x7f343800c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 1 -- 172.21.15.37:0/4014719262 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438004ca0 con 0x7f34500a2750 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 10 monclient: dump: 2024-09-16T07:22:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.765 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.765 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.765 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.765 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.766 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.766 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.766 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.766 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.766 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 10 monclient: _finish_auth 0 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 5 monclient: authenticate success, global_id 44147 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 1 -- 172.21.15.37:0/4014719262 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3438004e20 con 0x7f34500a2750 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient: _renew_subs 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 1 -- 172.21.15.37:0/4014719262 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3450072520 con 0x7f34500a2750 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 1 -- 172.21.15.37:0/4014719262 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438007310 con 0x7f34500a2750 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f3446ffd700 10 monclient: dump: 2024-09-16T07:22:59.767 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344d59a700 4 set_mon_vals no callback set 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient: _renew_subs 2024-09-16T07:22:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.749+0000 7f344f59e700 1 -- 172.21.15.37:0/4014719262 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3450124200 con 0x7f34500a2750 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.753+0000 7f344f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.753+0000 7f344f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.753+0000 7f344f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.753+0000 7f344f59e700 1 -- 172.21.15.37:0/4014719262 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3450124be0 con 0x7f34500a2750 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.753+0000 7f3446ffd700 1 -- 172.21.15.37:0/4014719262 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f3438007470 con 0x7f34500a2750 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.753+0000 7f3446ffd700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f34300355b0 0x7f3430037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.753+0000 7f3446ffd700 1 -- 172.21.15.37:0/4014719262 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f3438047770 con 0x7f34500a2750 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.753+0000 7f344e59c700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f34300355b0 0x7f3430037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.753+0000 7f344e59c700 10 monclient: get_auth_request con 0x7f34300355b0 auth_method 0 2024-09-16T07:22:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.753+0000 7f344e59c700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f34300355b0 0x7f3430037a70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3440003a30 tx=0x7f3440009270 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:22:59.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.761+0000 7f3446ffd700 1 -- 172.21.15.37:0/4014719262 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3450124be0 con 0x7f34500a2750 2024-09-16T07:22:59.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.761+0000 7f3446ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:22:59.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.761+0000 7f3446ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:22:59.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:22:59.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:23:00.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:00.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 -- 172.21.15.37:0/4014719262 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3450124be0 con 0x7f34500a2750 2024-09-16T07:23:00.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f3446ffd700 1 -- 172.21.15.37:0/4014719262 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f3450124be0 con 0x7f34500a2750 2024-09-16T07:23:00.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f3446ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:23:00.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f3446ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:23:00.001 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:23:00.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 -- 172.21.15.37:0/4014719262 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f34300355b0 msgr2=0x7f3430037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:00.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f34300355b0 0x7f3430037a70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3440003a30 tx=0x7f3440009270 comp rx=0 tx=0).stop 2024-09-16T07:23:00.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 10 monclient: shutdown 2024-09-16T07:23:00.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:00.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 -- 172.21.15.37:0/4014719262 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34500a2750 msgr2=0x7f34501c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:00.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34500a2750 0x7f34501c8bf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f343800c520 tx=0x7f343800c8e0 comp rx=0 tx=0).stop 2024-09-16T07:23:00.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 -- 172.21.15.37:0/4014719262 shutdown_connections 2024-09-16T07:23:00.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34500a31e0 0x7f3450120c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:00.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f34300355b0 0x7f3430037a70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:00.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34500a2750 0x7f34501c8bf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:00.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 --2- 172.21.15.37:0/4014719262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3450071880 0x7f34501c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:00.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 -- 172.21.15.37:0/4014719262 >> 172.21.15.37:0/4014719262 conn(0x7f34500a3ea0 msgr2=0x7f345009ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:00.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:22:59.997+0000 7f344f59e700 1 -- 172.21.15.37:0/4014719262 shutdown_connections 2024-09-16T07:23:00.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:00.001+0000 7f344f59e700 1 -- 172.21.15.37:0/4014719262 wait complete. 2024-09-16T07:23:03.018 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:23:03.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient: get_monmap_and_config 2024-09-16T07:23:03.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient: build_initial_monmap 2024-09-16T07:23:03.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient: monmap: 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:23:03.136047+0000 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:23:03.136047+0000 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:23:03.136 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 1 Processor -- start 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 1 -- start start 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient: init 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ff40a2750 addr v1:172.21.15.37:6789/0 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ff40a3240 addr v1:172.21.15.143:6789/0 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8ff409d580 0x7f8ff413ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient(hunting): picked mon.noname-a con 0x7f8ff409d580 addr v2:172.21.15.37:3300/0 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff409c6c0 con 0x7f8ff40a2750 2024-09-16T07:23:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff409c9a0 con 0x7f8ff40a3240 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff409b130 con 0x7f8ff409d580 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 10 monclient(hunting): authenticate will time out at 1103.621874s 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9164700 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f8ff40a2750 legacy=0x7f8ff40a2b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9965700 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f8ff409d580 msgr2=0x7f8ff413ab90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9965700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8ff409d580 0x7f8ff413ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff8963700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8ff40a3240 0x7f8ff4137460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.37:33814/0 (socket says 172.21.15.37:33814) 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff8963700 1 -- 172.21.15.37:0/1330971184 learned_addr learned my addr 172.21.15.37:0/1330971184 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007082472 0 0) 0x7f8ff409c9a0 con 0x7f8ff40a3240 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): _negotiate 2024-09-16T07:23:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): my global_id is 5767 2024-09-16T07:23:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe00035d0 con 0x7f8ff40a3240 2024-09-16T07:23:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2819084152 0 0) 0x7f8fe00035d0 con 0x7f8ff40a3240 2024-09-16T07:23:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 >> v2:172.21.15.37:3300/0 conn(0x7f8ff409d580 msgr2=0x7f8ff413ab90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:23:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 --2- 172.21.15.37:0/1330971184 >> v2:172.21.15.37:3300/0 conn(0x7f8ff409d580 0x7f8ff413ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 >> v1:172.21.15.37:6789/0 conn(0x7f8ff40a2750 legacy=0x7f8ff40a2b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:23:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient: _finish_hunting 0 2024-09-16T07:23:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 monclient: found mon.noname-c-legacy 2024-09-16T07:23:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.143:6789/0 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff409c3e0 con 0x7f8ff40a3240 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient: _finish_auth 0 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient: _check_auth_tickets 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 5 monclient: authenticate success, global_id 5767 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 120013200 0 0) 0x7f8ff0001670 con 0x7f8ff40a3240 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient: dump: 2024-09-16T07:23:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] but i'm connected to v1:172.21.15.143:6789/0, reconnecting 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 >> v1:172.21.15.143:6789/0 conn(0x7f8ff40a3240 legacy=0x7f8ff4137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:03.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 --2- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fe0003a20 0x7f8fe0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): picked mon.c con 0x7f8fe0003a20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 --2- 172.21.15.37:0/1330971184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe0004870 0x7f8fe0004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): picked mon.b con 0x7f8fe0004870 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 --2- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fe0005430 0x7f8fe000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): picked mon.a con 0x7f8fe0005430 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff409b130 con 0x7f8fe0005430 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe000c2d0 con 0x7f8fe0004870 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:03.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe000c490 con 0x7f8fe0003a20 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8fe7fff700 10 monclient(hunting): _finish_auth 0 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9164700 1 --2- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fe0003a20 0x7f8fe0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9164700 10 monclient(hunting): get_auth_request con 0x7f8fe0003a20 auth_method 0 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9164700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9164700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9164700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ffa166700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9965700 1 -- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fe0005430 msgr2=0x7f8fe000ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9164700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9164700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9965700 1 --2- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fe0005430 0x7f8fe000ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:23:03.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.133+0000 7f8ff9164700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 10 monclient(hunting): handle_auth_done global_id 5767 payload 293 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 1 -- 172.21.15.37:0/1330971184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe0004870 msgr2=0x7f8fe0004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 1 --2- 172.21.15.37:0/1330971184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe0004870 0x7f8fe0004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 1 -- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fe0005430 msgr2=0x7f8fe000ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 1 --2- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fe0005430 0x7f8fe000ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 10 monclient: _finish_hunting 0 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 1 monclient: found mon.c 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 1 -- 172.21.15.37:0/1330971184 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe00035d0 con 0x7f8fe0003a20 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 10 monclient: _finish_auth 0 2024-09-16T07:23:03.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 10 monclient: _check_auth_tickets 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 -- 172.21.15.37:0/1330971184 reap_dead start 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 1 --2- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fe0003a20 0x7f8fe0003e20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80099f0 tx=0x7f8fe00058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ac3af5c9f4e0776 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe800bd60 con 0x7f8fe0003a20 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8fe7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8fe7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8fe7fff700 10 monclient: dump: 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:03.149 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe8003a60 con 0x7f8fe0003a20 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8fe7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient: get_monmap_and_config success 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 4 set_mon_vals no callback set 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8fe7fff700 1 -- 172.21.15.37:0/1330971184 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8003e10 con 0x7f8fe0003a20 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8fe7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:03.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8fe7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8fe7fff700 10 monclient: dump: 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:03.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient: shutdown 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 -- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fe0003a20 msgr2=0x7f8fe0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 --2- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fe0003a20 0x7f8fe0003e20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8fe80099f0 tx=0x7f8fe00058a0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 -- 172.21.15.37:0/1330971184 shutdown_connections 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 --2- 172.21.15.37:0/1330971184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fe0003a20 0x7f8fe0003e20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 -- 172.21.15.37:0/1330971184 >> 172.21.15.37:0/1330971184 conn(0x7f8ff40a3ea0 msgr2=0x7f8ff40a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 -- 172.21.15.37:0/1330971184 shutdown_connections 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 -- 172.21.15.37:0/1330971184 wait complete. 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient: build_initial_monmap 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient: monmap: 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:23:03.152 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:23:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:23:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:23:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:23:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:23:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:23:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:23:03.153 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 Processor -- start 2024-09-16T07:23:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 -- start start 2024-09-16T07:23:03.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient: init 2024-09-16T07:23:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:23:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe0003a20 0x7f8ff41ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient(hunting): picked mon.noname-b con 0x7f8fe0003a20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:23:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ff41caa20 0x7f8ff4121640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient(hunting): picked mon.noname-a con 0x7f8ff41caa20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:23:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ff41cae60 0x7f8ff4121b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient(hunting): picked mon.noname-c con 0x7f8ff41cae60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40724c0 con 0x7f8ff41caa20 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff4123e60 con 0x7f8fe0003a20 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff41322d0 con 0x7f8ff41cae60 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ffa166700 10 monclient(hunting): authenticate will time out at 1103.626997s 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff8963700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ff41caa20 msgr2=0x7f8ff4121640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff8963700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ff41caa20 0x7f8ff4121640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ff41cae60 0x7f8ff4121b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:03.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ff41cae60 0x7f8ff4121b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49788/0 (socket says 172.21.15.37:49788) 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 -- 172.21.15.37:0/2245222720 learned_addr learned my addr 172.21.15.37:0/2245222720 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 10 monclient(hunting): get_auth_request con 0x7f8ff41cae60 auth_method 0 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe0003a20 0x7f8ff41ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 10 monclient(hunting): get_auth_request con 0x7f8fe0003a20 auth_method 0 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9164700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 10 monclient(hunting): handle_auth_done global_id 44150 payload 824 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 -- 172.21.15.37:0/2245222720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe0003a20 msgr2=0x7f8ff41ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe0003a20 0x7f8ff41ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 -- 172.21.15.37:0/2245222720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ff41caa20 msgr2=0x7f8ff4121640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ff41caa20 0x7f8ff4121640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 10 monclient: _finish_hunting 0 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 monclient: found mon.noname-c 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 -- 172.21.15.37:0/2245222720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff4070e30 con 0x7f8ff41cae60 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.137+0000 7f8ff9965700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ff41cae60 0x7f8ff4121b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4069c00 tx=0x7f8fec009e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 1 -- 172.21.15.37:0/2245222720 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00ce50 con 0x7f8ff41cae60 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 10 monclient: dump: 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 10 monclient: _finish_auth 0 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 5 monclient: authenticate success, global_id 44150 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 1 -- 172.21.15.37:0/2245222720 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec016430 con 0x7f8ff41cae60 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 1 -- 172.21.15.37:0/2245222720 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec0167e0 con 0x7f8ff41cae60 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 10 monclient: _renew_subs 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe77fe700 4 set_mon_vals no callback set 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 1 -- 172.21.15.37:0/2245222720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff4123290 con 0x7f8ff41cae60 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:23:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 10 monclient: dump: 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 10 monclient: _renew_subs 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 1 -- 172.21.15.37:0/2245222720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff41255f0 con 0x7f8ff41cae60 2024-09-16T07:23:03.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 1 -- 172.21.15.37:0/2245222720 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f8fec012020 con 0x7f8ff41cae60 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ffa166700 1 -- 172.21.15.37:0/2245222720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff4125150 con 0x7f8ff41cae60 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f8fd80355b0 0x7f8fd8037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8fe5ffb700 1 -- 172.21.15.37:0/2245222720 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f8fec0477c0 con 0x7f8ff41cae60 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ff9164700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f8fd80355b0 0x7f8fd8037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ff9164700 10 monclient: get_auth_request con 0x7f8fd80355b0 auth_method 0 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.141+0000 7f8ff9164700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f8fd80355b0 0x7f8fd8037a70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009fd0 tx=0x7f8fe8011000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.149+0000 7f8fe5ffb700 1 -- 172.21.15.37:0/2245222720 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8fec013d40 con 0x7f8ff41cae60 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.149+0000 7f8fe5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.149+0000 7f8fe5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:23:03.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.381+0000 7f8ffa166700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:23:03.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.381+0000 7f8ffa166700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:23:03.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.381+0000 7f8ffa166700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:03.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.381+0000 7f8ffa166700 1 -- 172.21.15.37:0/2245222720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8ff40a2b30 con 0x7f8ff41cae60 2024-09-16T07:23:03.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.381+0000 7f8fe5ffb700 1 -- 172.21.15.37:0/2245222720 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f8fec0132e0 con 0x7f8ff41cae60 2024-09-16T07:23:03.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.381+0000 7f8fe5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:23:03.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.381+0000 7f8fe5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:23:03.385 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:23:03.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 -- 172.21.15.37:0/2245222720 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f8fd80355b0 msgr2=0x7f8fd8037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:03.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f8fd80355b0 0x7f8fd8037a70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009fd0 tx=0x7f8fe8011000 comp rx=0 tx=0).stop 2024-09-16T07:23:03.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 10 monclient: shutdown 2024-09-16T07:23:03.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:03.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 -- 172.21.15.37:0/2245222720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ff41cae60 msgr2=0x7f8ff4121b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:03.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ff41cae60 0x7f8ff4121b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4069c00 tx=0x7f8fec009e50 comp rx=0 tx=0).stop 2024-09-16T07:23:03.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 -- 172.21.15.37:0/2245222720 shutdown_connections 2024-09-16T07:23:03.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f8fd80355b0 0x7f8fd8037a70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ff41cae60 0x7f8ff4121b80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ff41caa20 0x7f8ff4121640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 --2- 172.21.15.37:0/2245222720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe0003a20 0x7f8ff41ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:03.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 -- 172.21.15.37:0/2245222720 >> 172.21.15.37:0/2245222720 conn(0x7f8ff40a3ea0 msgr2=0x7f8ff409ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:03.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 -- 172.21.15.37:0/2245222720 shutdown_connections 2024-09-16T07:23:03.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:03.385+0000 7f8ffa166700 1 -- 172.21.15.37:0/2245222720 wait complete. 2024-09-16T07:23:06.400 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:23:06.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.516+0000 7f331db82700 10 monclient: get_monmap_and_config 2024-09-16T07:23:06.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.516+0000 7f331db82700 10 monclient: build_initial_monmap 2024-09-16T07:23:06.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.516+0000 7f331db82700 10 monclient: monmap: 2024-09-16T07:23:06.522 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:23:06.522 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:06.522 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:23:06.522550+0000 2024-09-16T07:23:06.522 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:23:06.522550+0000 2024-09-16T07:23:06.523 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:23:06.523 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:06.523 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:23:06.523 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:23:06.523 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:23:06.523 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:23:06.523 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:23:06.523 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:06.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.516+0000 7f331db82700 1 Processor -- start 2024-09-16T07:23:06.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.516+0000 7f331db82700 1 -- start start 2024-09-16T07:23:06.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.516+0000 7f331db82700 10 monclient: init 2024-09-16T07:23:06.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:06.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T07:23:06.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a2750 0x7f33180a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:06.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient(hunting): picked mon.noname-b con 0x7f33180a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33180a3070 addr v1:172.21.15.37:6789/0 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3318133730 0x7f331813b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient(hunting): picked mon.noname-c con 0x7f3318133730 addr v2:172.21.15.143:3300/0 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3318099be0 con 0x7f33180a3070 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331809de40 con 0x7f3318133730 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331809d530 con 0x7f33180a2750 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient(hunting): authenticate will time out at 1107.008395s 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f3317fff700 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f33180a3070 legacy=0x7f3318133020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a2750 0x7f33180a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:06.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a2750 0x7f33180a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49796/0 (socket says 172.21.15.37:49796) 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 1 -- 172.21.15.37:0/2200822413 learned_addr learned my addr 172.21.15.37:0/2200822413 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 10 monclient(hunting): get_auth_request con 0x7f33180a2750 auth_method 0 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331d381700 1 --2- 172.21.15.37:0/2200822413 >> v2:172.21.15.143:3300/0 conn(0x7f3318133730 0x7f331813b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331d381700 10 monclient(hunting): get_auth_request con 0x7f3318133730 auth_method 0 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331d381700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331d381700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:06.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331d381700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 10 monclient(hunting): handle_auth_done global_id 44153 payload 274 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 1 -- 172.21.15.37:0/2200822413 >> v2:172.21.15.143:3300/0 conn(0x7f3318133730 msgr2=0x7f331813b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 1 --2- 172.21.15.37:0/2200822413 >> v2:172.21.15.143:3300/0 conn(0x7f3318133730 0x7f331813b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 1 -- 172.21.15.37:0/2200822413 >> v1:172.21.15.37:6789/0 conn(0x7f33180a3070 legacy=0x7f3318133020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 10 monclient: _finish_hunting 0 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 1 monclient: found mon.noname-b 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 1 -- 172.21.15.37:0/2200822413 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331809a020 con 0x7f33180a2750 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331cb80700 1 --2- 172.21.15.37:0/2200822413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a2750 0x7f33180a2b30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3308009910 tx=0x7f3308009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6df2a04070250d2b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 1 -- 172.21.15.37:0/2200822413 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308004230 con 0x7f33180a2750 2024-09-16T07:23:06.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 10 monclient: dump: 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 10 monclient: _finish_auth 0 2024-09-16T07:23:06.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 10 monclient: _check_auth_tickets 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 1 -- 172.21.15.37:0/2200822413 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3308004390 con 0x7f33180a2750 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 5 monclient: authenticate success, global_id 44153 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 1 -- 172.21.15.37:0/2200822413 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308009e60 con 0x7f33180a2750 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f33177fe700 10 monclient: dump: 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:06.531 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient: get_monmap_and_config success 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 4 set_mon_vals no callback set 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 10 monclient: shutdown 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 -- 172.21.15.37:0/2200822413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a2750 msgr2=0x7f33180a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 --2- 172.21.15.37:0/2200822413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a2750 0x7f33180a2b30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3308009910 tx=0x7f3308009c20 comp rx=0 tx=0).stop 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 -- 172.21.15.37:0/2200822413 shutdown_connections 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 --2- 172.21.15.37:0/2200822413 >> v2:172.21.15.143:3300/0 conn(0x7f3318133730 0x7f331813b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:06.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 --2- 172.21.15.37:0/2200822413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a2750 0x7f33180a2b30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:06.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 -- 172.21.15.37:0/2200822413 >> 172.21.15.37:0/2200822413 conn(0x7f33180a3ea0 msgr2=0x7f33180a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:06.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 -- 172.21.15.37:0/2200822413 shutdown_connections 2024-09-16T07:23:06.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.520+0000 7f331db82700 1 -- 172.21.15.37:0/2200822413 wait complete. 2024-09-16T07:23:06.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient: build_initial_monmap 2024-09-16T07:23:06.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient: monmap: 2024-09-16T07:23:06.537 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:23:06.537 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:23:06.537 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:23:06.537 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:23:06.537 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 1 Processor -- start 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 1 -- start start 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient: init 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33180a2750 0x7f33181323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient(hunting): picked mon.noname-a con 0x7f33180a2750 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:23:06.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a3070 0x7f3318132900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient(hunting): picked mon.noname-c con 0x7f33180a3070 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3318133730 0x7f331811bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient(hunting): picked mon.noname-b con 0x7f3318133730 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33181c8f00 con 0x7f33180a2750 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331811f4e0 con 0x7f3318133730 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3318132e40 con 0x7f33180a3070 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331db82700 10 monclient(hunting): authenticate will time out at 1107.014738s 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331cb80700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33180a2750 msgr2=0x7f33181323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331cb80700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33180a2750 0x7f33181323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:23:06.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a3070 0x7f3318132900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a3070 0x7f3318132900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49810/0 (socket says 172.21.15.37:49810) 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 1 -- 172.21.15.37:0/1744166289 learned_addr learned my addr 172.21.15.37:0/1744166289 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 10 monclient(hunting): get_auth_request con 0x7f33180a3070 auth_method 0 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331d381700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3318133730 0x7f331811bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331d381700 10 monclient(hunting): get_auth_request con 0x7f3318133730 auth_method 0 2024-09-16T07:23:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331d381700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331d381700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331d381700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 10 monclient(hunting): handle_auth_done global_id 44156 payload 824 2024-09-16T07:23:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 1 -- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3318133730 msgr2=0x7f331811bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3318133730 0x7f331811bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 1 -- 172.21.15.37:0/1744166289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33180a2750 msgr2=0x7f33181323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:23:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33180a2750 0x7f33181323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 10 monclient: _finish_hunting 0 2024-09-16T07:23:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 1 monclient: found mon.noname-c 2024-09-16T07:23:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 1 -- 172.21.15.37:0/1744166289 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3308009570 con 0x7f33180a3070 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f331d381700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3318133730 0x7f331811bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f3317fff700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a3070 0x7f3318132900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f330c00b5a0 tx=0x7f330c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.524+0000 7f33157fa700 1 -- 172.21.15.37:0/1744166289 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c003e70 con 0x7f33180a3070 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 10 monclient: dump: 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 10 monclient: _finish_auth 0 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 10 monclient: _check_auth_tickets 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 5 monclient: authenticate success, global_id 44156 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 1 -- 172.21.15.37:0/1744166289 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c005070 con 0x7f33180a3070 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:23:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 10 monclient: _renew_subs 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 1 -- 172.21.15.37:0/1744166289 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331811fd80 con 0x7f33180a3070 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 1 -- 172.21.15.37:0/1744166289 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c005440 con 0x7f33180a3070 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f3316ffd700 4 set_mon_vals no callback set 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 10 monclient: dump: 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 10 monclient: _renew_subs 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 1 -- 172.21.15.37:0/1744166289 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331811faa0 con 0x7f33180a3070 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331db82700 1 -- 172.21.15.37:0/1744166289 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331811f620 con 0x7f33180a3070 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 1 -- 172.21.15.37:0/1744166289 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f330c0055a0 con 0x7f33180a3070 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33040355b0 0x7f3304037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f33157fa700 1 -- 172.21.15.37:0/1744166289 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f330c046f20 con 0x7f33180a3070 2024-09-16T07:23:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331cb80700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33040355b0 0x7f3304037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331cb80700 10 monclient: get_auth_request con 0x7f33040355b0 auth_method 0 2024-09-16T07:23:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.528+0000 7f331cb80700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33040355b0 0x7f3304037a70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3308009910 tx=0x7f3308008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.540+0000 7f33157fa700 1 -- 172.21.15.37:0/1744166289 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f330c078db0 con 0x7f33180a3070 2024-09-16T07:23:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.540+0000 7f33157fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.540+0000 7f33157fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:23:06.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.772+0000 7f331db82700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:23:06.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.772+0000 7f331db82700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:23:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.772+0000 7f331db82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.772+0000 7f331db82700 1 -- 172.21.15.37:0/1744166289 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f331809cd00 con 0x7f33180a3070 2024-09-16T07:23:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f33157fa700 1 -- 172.21.15.37:0/1744166289 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1544 (secure 0 0 0) 0x7f330c016070 con 0x7f33180a3070 2024-09-16T07:23:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f33157fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:23:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f33157fa700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:23:06.779 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:23:06.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 -- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33040355b0 msgr2=0x7f3304037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:06.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33040355b0 0x7f3304037a70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3308009910 tx=0x7f3308008030 comp rx=0 tx=0).stop 2024-09-16T07:23:06.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 10 monclient: shutdown 2024-09-16T07:23:06.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:06.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 -- 172.21.15.37:0/1744166289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a3070 msgr2=0x7f3318132900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:06.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a3070 0x7f3318132900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f330c00b5a0 tx=0x7f330c00b960 comp rx=0 tx=0).stop 2024-09-16T07:23:06.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 -- 172.21.15.37:0/1744166289 shutdown_connections 2024-09-16T07:23:06.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3318133730 0x7f331811bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:06.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f33040355b0 0x7f3304037a70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:06.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33180a3070 0x7f3318132900 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:06.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 --2- 172.21.15.37:0/1744166289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33180a2750 0x7f33181323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:06.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 -- 172.21.15.37:0/1744166289 >> 172.21.15.37:0/1744166289 conn(0x7f33180a3ea0 msgr2=0x7f331809f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:06.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 -- 172.21.15.37:0/1744166289 shutdown_connections 2024-09-16T07:23:06.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:06.776+0000 7f331db82700 1 -- 172.21.15.37:0/1744166289 wait complete. 2024-09-16T07:23:09.795 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.908+0000 7fadd89bc700 10 monclient: get_monmap_and_config 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.908+0000 7fadd89bc700 10 monclient: build_initial_monmap 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.908+0000 7fadd89bc700 10 monclient: monmap: 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:23:09.914237+0000 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:23:09.914237+0000 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:23:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:23:09.915 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:23:09.915 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:23:09.915 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:09.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.908+0000 7fadd89bc700 1 Processor -- start 2024-09-16T07:23:09.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.908+0000 7fadd89bc700 1 -- start start 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.908+0000 7fadd89bc700 10 monclient: init 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fadd40a2750 addr v1:172.21.15.143:6789/0 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fadd40a3240 addr v1:172.21.15.37:6789/0 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 10 monclient(hunting): picked mon.noname-b con 0x7fadd409d580 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd409c6c0 con 0x7fadd40a3240 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd409c9a0 con 0x7fadd40a2750 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd409b130 con 0x7fadd409d580 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd89bc700 10 monclient(hunting): authenticate will time out at 1110.400352s 2024-09-16T07:23:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadcbfff700 1 -- >> v1:172.21.15.37:6789/0 conn(0x7fadd40a3240 legacy=0x7fadd4137460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49824/0 (socket says 172.21.15.37:49824) 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 1 -- 172.21.15.37:0/2088301777 learned_addr learned my addr 172.21.15.37:0/2088301777 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 10 monclient(hunting): get_auth_request con 0x7fadd409d580 auth_method 0 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 1 -- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 msgr2=0x7fadd413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 1 -- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 msgr2=0x7fadd413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:23:09.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:09.912+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:23:10.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.116+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 >> v1:172.21.15.37:6789/0 conn(0x7fadd40a3240 legacy=0x7fadd4137460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:23:10.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.116+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:10.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.116+0000 7fadd359e700 10 monclient(hunting): get_auth_request con 0x7fadd409d580 auth_method 0 2024-09-16T07:23:10.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.116+0000 7fadd359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:10.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.116+0000 7fadd359e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:10.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.116+0000 7fadd359e700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:23:10.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.116+0000 7fadd359e700 1 -- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 msgr2=0x7fadd413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:23:10.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.116+0000 7fadd359e700 1 -- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 msgr2=0x7fadd413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:23:10.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.116+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:23:10.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.116+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:23:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.520+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 >> v1:172.21.15.37:6789/0 conn(0x7fadd40a3240 legacy=0x7fadd4137460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:23:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.520+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.520+0000 7fadd359e700 10 monclient(hunting): get_auth_request con 0x7fadd409d580 auth_method 0 2024-09-16T07:23:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.520+0000 7fadd359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.520+0000 7fadd359e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.520+0000 7fadd359e700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:23:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.520+0000 7fadd359e700 1 -- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 msgr2=0x7fadd413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:23:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.520+0000 7fadd359e700 1 -- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 msgr2=0x7fadd413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:23:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.520+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:23:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:10.520+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:23:11.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:11.324+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 >> v1:172.21.15.37:6789/0 conn(0x7fadd40a3240 legacy=0x7fadd4137460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:23:11.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:11.324+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:11.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:11.324+0000 7fadd359e700 10 monclient(hunting): get_auth_request con 0x7fadd409d580 auth_method 0 2024-09-16T07:23:11.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:11.324+0000 7fadd359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:11.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:11.324+0000 7fadd359e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:11.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:11.324+0000 7fadd359e700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:23:11.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:11.324+0000 7fadd359e700 1 -- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 msgr2=0x7fadd413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:23:11.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:11.324+0000 7fadd359e700 1 -- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 msgr2=0x7fadd413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:23:11.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:11.324+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:23:11.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:11.324+0000 7fadd359e700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T07:23:12.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 10 monclient(hunting): tick 2024-09-16T07:23:12.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 10 monclient(hunting): _check_auth_tickets 2024-09-16T07:23:12.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 1 monclient(hunting): continuing hunt 2024-09-16T07:23:12.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 10 monclient(hunting): _reopen_session rank -1 2024-09-16T07:23:12.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 1 -- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 msgr2=0x7fadd413ab90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:23:12.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd409d580 0x7fadd413ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 1 -- 172.21.15.37:0/2088301777 >> v1:172.21.15.143:6789/0 conn(0x7fadd40a2750 legacy=0x7fadd40a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 1 -- 172.21.15.37:0/2088301777 >> v1:172.21.15.37:6789/0 conn(0x7fadd40a3240 legacy=0x7fadd4137460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 10 monclient: _add_conns ranks=[3,4] 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 0x7fadb4005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 10 monclient(hunting): picked mon.noname-a con 0x7fadb4003780 addr v2:172.21.15.37:3300/0 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 10 monclient(hunting): picked mon.noname-c con 0x7fadb40062f0 addr v2:172.21.15.143:3300/0 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 1 -- 172.21.15.37:0/2088301777 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd409b130 con 0x7fadb4003780 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 1 -- 172.21.15.37:0/2088301777 --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb4008e70 con 0x7fadb40062f0 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd159a700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:12.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd2d9d700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 msgr2=0x7fadb4005c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:12.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadd2d9d700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 0x7fadb4005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:23:12.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:12.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadcbfff700 10 monclient(hunting): get_auth_request con 0x7fadb40062f0 auth_method 0 2024-09-16T07:23:12.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:12.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:12.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:12.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 msgr2=0x7fadb40087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:23:12.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 msgr2=0x7fadb40087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:23:12.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:23:12.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:12.912+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:23:13.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadd2d9d700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 msgr2=0x7fadb4005c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadd2d9d700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 0x7fadb4005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:23:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadcbfff700 10 monclient(hunting): get_auth_request con 0x7fadb40062f0 auth_method 0 2024-09-16T07:23:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadcbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:23:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 msgr2=0x7fadb40087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T07:23:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 msgr2=0x7fadb40087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:23:13.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:23:13.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.116+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T07:23:13.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadd2d9d700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 msgr2=0x7fadb4005c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadd2d9d700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 0x7fadb4005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:23:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadcbfff700 10 monclient(hunting): get_auth_request con 0x7fadb40062f0 auth_method 0 2024-09-16T07:23:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadcbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:23:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 msgr2=0x7fadb40087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T07:23:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 msgr2=0x7fadb40087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:23:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:23:13.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:13.520+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T07:23:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd2d9d700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 msgr2=0x7fadb4005c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd2d9d700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 0x7fadb4005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T07:23:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.143:3300/0 conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 10 monclient(hunting): get_auth_request con 0x7fadb40062f0 auth_method 0 2024-09-16T07:23:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:23:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 10 monclient(hunting): handle_auth_done global_id 5779 payload 274 2024-09-16T07:23:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 msgr2=0x7fadb4005c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:23:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> v2:172.21.15.37:3300/0 conn(0x7fadb4003780 0x7fadb4005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 10 monclient: _finish_hunting 0 2024-09-16T07:23:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 1 monclient: found mon.noname-c 2024-09-16T07:23:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:23:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 1 -- 172.21.15.37:0/2088301777 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd409c3e0 con 0x7fadb40062f0 2024-09-16T07:23:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadcbfff700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 0x7fadb40087b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fadb40062c0 tx=0x7fadc400b6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cd8c838c51f6ab8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 1 -- 172.21.15.37:0/2088301777 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4016610 con 0x7fadb40062f0 2024-09-16T07:23:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:23:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 10 monclient: dump: 2024-09-16T07:23:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.331 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 10 monclient: _finish_auth 0 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 10 monclient: _check_auth_tickets 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 1 -- 172.21.15.37:0/2088301777 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc4011050 con 0x7fadb40062f0 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd89bc700 5 monclient: authenticate success, global_id 5779 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd89bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 1 -- 172.21.15.37:0/2088301777 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4016990 con 0x7fadb40062f0 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd89bc700 10 monclient: get_monmap_and_config success 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd89bc700 4 set_mon_vals no callback set 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:23:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.324+0000 7fadd259c700 10 monclient: dump: 2024-09-16T07:23:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.333 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 10 monclient: shutdown 2024-09-16T07:23:14.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:14.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 1 -- 172.21.15.37:0/2088301777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 msgr2=0x7fadb40087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 0x7fadb40087b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fadb40062c0 tx=0x7fadc400b6d0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd359e700 1 -- 172.21.15.37:0/2088301777 reap_dead start 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 1 -- 172.21.15.37:0/2088301777 shutdown_connections 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 1 --2- 172.21.15.37:0/2088301777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 0x7fadb40087b0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 1 -- 172.21.15.37:0/2088301777 >> 172.21.15.37:0/2088301777 conn(0x7fadd40a3ea0 msgr2=0x7fadd40a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 1 -- 172.21.15.37:0/2088301777 shutdown_connections 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 1 -- 172.21.15.37:0/2088301777 wait complete. 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 10 monclient: build_initial_monmap 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 10 monclient: monmap: 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:23:14.340 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 1 Processor -- start 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 1 -- start start 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.328+0000 7fadd89bc700 10 monclient: init 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 0x7fadd41ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient(hunting): picked mon.noname-b con 0x7fadb40062f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fadd41caa20 0x7fadd4120680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient(hunting): picked mon.noname-a con 0x7fadd41caa20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd41cae80 0x7fadd4120bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient(hunting): picked mon.noname-c con 0x7fadd41cae80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd4123d40 con 0x7fadd41caa20 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd41245e0 con 0x7fadb40062f0 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd409fac0 con 0x7fadd41cae80 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient(hunting): authenticate will time out at 1114.820219s 2024-09-16T07:23:14.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcbfff700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fadd41caa20 msgr2=0x7fadd4120680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcbfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fadd41caa20 0x7fadd4120680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd359e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd41cae80 0x7fadd4120bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 0x7fadd41ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 0x7fadd41ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.37:39714/0 (socket says 172.21.15.37:39714) 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd359e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd41cae80 0x7fadd4120bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51480/0 (socket says 172.21.15.37:51480) 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 1 -- 172.21.15.37:0/1158212640 learned_addr learned my addr 172.21.15.37:0/1158212640 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 10 monclient(hunting): get_auth_request con 0x7fadb40062f0 auth_method 0 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd359e700 10 monclient(hunting): get_auth_request con 0x7fadd41cae80 auth_method 0 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd359e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 10 monclient(hunting): handle_auth_done global_id 5782 payload 824 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 1 -- 172.21.15.37:0/1158212640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd41cae80 msgr2=0x7fadd4120bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd41cae80 0x7fadd4120bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 1 -- 172.21.15.37:0/1158212640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fadd41caa20 msgr2=0x7fadd4120680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:23:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fadd41caa20 0x7fadd4120680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 10 monclient: _finish_hunting 0 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 1 monclient: found mon.noname-b 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.143:3300/0 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 1 -- 172.21.15.37:0/1158212640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc4009a20 con 0x7fadb40062f0 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd2d9d700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 0x7fadd41ca4e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00c730 tx=0x7fadbc00ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 1 -- 172.21.15.37:0/1158212640 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc005c80 con 0x7fadb40062f0 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 10 monclient: dump: 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.345 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 10 monclient: _finish_auth 0 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 5 monclient: authenticate success, global_id 5782 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient: _renew_subs 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:23:14.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 1 -- 172.21.15.37:0/1158212640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd40724c0 con 0x7fadb40062f0 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient: _renew_subs 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd89bc700 1 -- 172.21.15.37:0/1158212640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd4122bd0 con 0x7fadb40062f0 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 1 -- 172.21.15.37:0/1158212640 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadbc005de0 con 0x7fadb40062f0 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 1 -- 172.21.15.37:0/1158212640 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadbc007310 con 0x7fadb40062f0 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadcb7fe700 10 monclient: dump: 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.347 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.332+0000 7fadd1d9b700 4 set_mon_vals no callback set 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.336+0000 7fadd89bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.336+0000 7fadd89bc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.336+0000 7fadd89bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.336+0000 7fadd89bc700 1 -- 172.21.15.37:0/1158212640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb8005100 con 0x7fadb40062f0 2024-09-16T07:23:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.336+0000 7fadcb7fe700 1 -- 172.21.15.37:0/1158212640 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fadbc007470 con 0x7fadb40062f0 2024-09-16T07:23:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.336+0000 7fadcb7fe700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fadb4036410 0x7fadb40388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.336+0000 7fadcb7fe700 1 -- 172.21.15.37:0/1158212640 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fadbc046e80 con 0x7fadb40062f0 2024-09-16T07:23:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.336+0000 7fadcbfff700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fadb4036410 0x7fadb40388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.336+0000 7fadcbfff700 10 monclient: get_auth_request con 0x7fadb4036410 auth_method 0 2024-09-16T07:23:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.336+0000 7fadcbfff700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fadb4036410 0x7fadb40388d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fadd4121e80 tx=0x7fadc400d030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.348+0000 7fadcb7fe700 1 -- 172.21.15.37:0/1158212640 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fadbc019ab0 con 0x7fadb40062f0 2024-09-16T07:23:14.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.348+0000 7fadcb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:14.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.348+0000 7fadcb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:23:14.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.588+0000 7fadd89bc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:23:14.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.588+0000 7fadd89bc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:23:14.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.588+0000 7fadd89bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.143:3300/0 2024-09-16T07:23:14.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.588+0000 7fadd89bc700 1 -- 172.21.15.37:0/1158212640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fadb80053b0 con 0x7fadb40062f0 2024-09-16T07:23:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.588+0000 7fadcb7fe700 1 -- 172.21.15.37:0/1158212640 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1537 (secure 0 0 0) 0x7fadbc017020 con 0x7fadb40062f0 2024-09-16T07:23:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.588+0000 7fadcb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:23:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.588+0000 7fadcb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:23:14.593 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:23:14.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 -- 172.21.15.37:0/1158212640 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fadb4036410 msgr2=0x7fadb40388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fadb4036410 0x7fadb40388d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fadd4121e80 tx=0x7fadc400d030 comp rx=0 tx=0).stop 2024-09-16T07:23:14.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 10 monclient: shutdown 2024-09-16T07:23:14.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:14.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 -- 172.21.15.37:0/1158212640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 msgr2=0x7fadd41ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 0x7fadd41ca4e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00c730 tx=0x7fadbc00ca40 comp rx=0 tx=0).stop 2024-09-16T07:23:14.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 -- 172.21.15.37:0/1158212640 shutdown_connections 2024-09-16T07:23:14.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fadb4036410 0x7fadb40388d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fadd41cae80 0x7fadd4120bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fadd41caa20 0x7fadd4120680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 --2- 172.21.15.37:0/1158212640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadb40062f0 0x7fadd41ca4e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 -- 172.21.15.37:0/1158212640 >> 172.21.15.37:0/1158212640 conn(0x7fadd40a3ea0 msgr2=0x7fadd4131210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:14.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 -- 172.21.15.37:0/1158212640 shutdown_connections 2024-09-16T07:23:14.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.592+0000 7fadd89bc700 1 -- 172.21.15.37:0/1158212640 wait complete. 2024-09-16T07:23:14.608 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T07:23:14.608 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T07:23:14.608 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T07:23:14.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient: get_monmap_and_config 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient: build_initial_monmap 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient: monmap: 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:23:14.721018+0000 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:23:14.721018+0000 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:23:14.721 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:23:14.722 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:23:14.722 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 1 Processor -- start 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 1 -- start start 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient: init 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec0809e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient(hunting): picked mon.noname-b con 0x7fec080a1530 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fec080a1910 0x7fec080a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient(hunting): picked mon.noname-c con 0x7fec080a1910 addr v2:172.21.15.143:3300/0 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec08135300 addr v1:172.21.15.143:6789/0 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0806d880 con 0x7fec08135300 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec080a00d0 con 0x7fec080a1910 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0809df20 con 0x7fec080a1530 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec0de81700 10 monclient(hunting): authenticate will time out at 1115.207018s 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec0809e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec0809e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51490/0 (socket says 172.21.15.37:51490) 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 1 -- 172.21.15.37:0/2591300435 learned_addr learned my addr 172.21.15.37:0/2591300435 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 10 monclient(hunting): get_auth_request con 0x7fec080a1530 auth_method 0 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7febfeffd700 1 --2- 172.21.15.37:0/2591300435 >> v2:172.21.15.143:3300/0 conn(0x7fec080a1910 0x7fec080a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7febfeffd700 10 monclient(hunting): get_auth_request con 0x7fec080a1910 auth_method 0 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7febfeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7febfeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7febfeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 10 monclient(hunting): handle_auth_done global_id 44162 payload 274 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 1 -- 172.21.15.37:0/2591300435 >> v2:172.21.15.143:3300/0 conn(0x7fec080a1910 msgr2=0x7fec080a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 1 --2- 172.21.15.37:0/2591300435 >> v2:172.21.15.143:3300/0 conn(0x7fec080a1910 0x7fec080a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 1 -- 172.21.15.37:0/2591300435 >> v1:172.21.15.143:6789/0 conn(0x7fec08135300 legacy=0x7fec08135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 10 monclient: _finish_hunting 0 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 1 monclient: found mon.noname-b 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:14.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:23:14.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7fec077fe700 1 -- 172.21.15.37:0/2591300435 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0806beb0 con 0x7fec080a1530 2024-09-16T07:23:14.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.716+0000 7febfeffd700 1 --2- 172.21.15.37:0/2591300435 >> v2:172.21.15.143:3300/0 conn(0x7fec080a1910 0x7fec080a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:23:14.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec077fe700 1 --2- 172.21.15.37:0/2591300435 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec0809e6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7febf8009910 tx=0x7febf8009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2240e9581e2c21df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 1 -- 172.21.15.37:0/2591300435 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8004230 con 0x7fec080a1530 2024-09-16T07:23:14.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:23:14.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 10 monclient: dump: 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 10 monclient: _finish_auth 0 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:23:14.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 5 monclient: authenticate success, global_id 44162 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 1 -- 172.21.15.37:0/2591300435 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf8004390 con 0x7fec080a1530 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 10 monclient: get_monmap_and_config success 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 4 set_mon_vals no callback set 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 1 -- 172.21.15.37:0/2591300435 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8009e60 con 0x7fec080a1530 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec06ffd700 10 monclient: dump: 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.732 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 10 monclient: shutdown 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 1 -- 172.21.15.37:0/2591300435 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 msgr2=0x7fec0809e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 1 --2- 172.21.15.37:0/2591300435 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec0809e6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7febf8009910 tx=0x7febf8009c20 comp rx=0 tx=0).stop 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 1 -- 172.21.15.37:0/2591300435 shutdown_connections 2024-09-16T07:23:14.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 1 --2- 172.21.15.37:0/2591300435 >> v2:172.21.15.143:3300/0 conn(0x7fec080a1910 0x7fec080a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 1 --2- 172.21.15.37:0/2591300435 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec0809e6c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 1 -- 172.21.15.37:0/2591300435 >> 172.21.15.37:0/2591300435 conn(0x7fec080a61b0 msgr2=0x7fec080a2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 1 -- 172.21.15.37:0/2591300435 shutdown_connections 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 1 -- 172.21.15.37:0/2591300435 wait complete. 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 10 monclient: build_initial_monmap 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.720+0000 7fec0de81700 10 monclient: monmap: 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:23:14.734 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 1 Processor -- start 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 1 -- start start 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient: init 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec081c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient(hunting): picked mon.noname-c con 0x7fec080a1530 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec080a1910 0x7fec081345d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient(hunting): picked mon.noname-b con 0x7fec080a1910 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec08135300 0x7fec08134b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient(hunting): picked mon.noname-a con 0x7fec08135300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec081ca3c0 con 0x7fec08135300 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec08121cb0 con 0x7fec080a1910 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec08120800 con 0x7fec080a1530 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec0de81700 10 monclient(hunting): authenticate will time out at 1115.212793s 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec07fff700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec08135300 msgr2=0x7fec08134b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec07fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec08135300 0x7fec08134b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec081c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec081c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51494/0 (socket says 172.21.15.37:51494) 2024-09-16T07:23:14.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 1 -- 172.21.15.37:0/3775759286 learned_addr learned my addr 172.21.15.37:0/3775759286 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 10 monclient(hunting): get_auth_request con 0x7fec080a1530 auth_method 0 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7febfeffd700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec080a1910 0x7fec081345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7febfeffd700 10 monclient(hunting): get_auth_request con 0x7fec080a1910 auth_method 0 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7febfeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7febfeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7febfeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 10 monclient(hunting): handle_auth_done global_id 44165 payload 824 2024-09-16T07:23:14.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 1 -- 172.21.15.37:0/3775759286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec080a1910 msgr2=0x7fec081345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec080a1910 0x7fec081345d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 1 -- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec08135300 msgr2=0x7fec08134b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:23:14.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec08135300 0x7fec08134b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 10 monclient: _finish_hunting 0 2024-09-16T07:23:14.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 1 monclient: found mon.noname-c 2024-09-16T07:23:14.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:14.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:23:14.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 1 -- 172.21.15.37:0/3775759286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf8009570 con 0x7fec080a1530 2024-09-16T07:23:14.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.724+0000 7fec077fe700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec081c9e80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fec080a35a0 tx=0x7febf8004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf80036c0 con 0x7fec080a1530 2024-09-16T07:23:14.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec04ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec04ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:23:14.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec04ff9700 10 monclient: dump: 2024-09-16T07:23:14.741 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.741 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.741 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.741 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec04ff9700 10 monclient: _finish_auth 0 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec04ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec04ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec0de81700 5 monclient: authenticate success, global_id 44165 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec0de81700 10 monclient: _renew_subs 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec0de81700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec0de81700 1 -- 172.21.15.37:0/3775759286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec080708c0 con 0x7fec080a1530 2024-09-16T07:23:14.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec0de81700 10 monclient: _renew_subs 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec0de81700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.728+0000 7fec0de81700 1 -- 172.21.15.37:0/3775759286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec081219d0 con 0x7fec080a1530 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec0de81700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec0de81700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811dfe0 0x7fec0811e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec0de81700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec0de81700 1 -- 172.21.15.37:0/3775759286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec08067720 con 0x7fec0811dfe0 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf8003820 con 0x7fec080a1530 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7febfeffd700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811dfe0 0x7fec0811e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7febfeffd700 10 monclient: get_auth_request con 0x7fec0811dfe0 auth_method 0 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7febfeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7febfeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7febfeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec04ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8003b90 con 0x7fec080a1530 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7febfeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7febfeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7febfeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec04ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec04ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec04ff9700 10 monclient: dump: 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.744 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.732+0000 7fec067fc700 4 set_mon_vals no callback set 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7febfeffd700 10 monclient(hunting): handle_auth_done global_id 44168 payload 824 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7febfeffd700 1 -- 172.21.15.37:0/3775759286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec0806c3b0 con 0x7fec0811dfe0 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7febf8021070 con 0x7fec080a1530 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7febe80355e0 0x7febe8037aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 4738+0+0 (secure 0 0 0) 0x7febf8049450 con 0x7fec080a1530 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7febfeffd700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811dfe0 0x7fec0811e3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec0811f830 tx=0x7febf00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec07fff700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7febe80355e0 0x7febe8037aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec07fff700 10 monclient: get_auth_request con 0x7febe80355e0 auth_method 0 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf000c070 con 0x7fec0811dfe0 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 10 monclient: dump: 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.746 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.747 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7febf0009e60 con 0x7fec0811dfe0 2024-09-16T07:23:14.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:14.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:23:14.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811dfe0 msgr2=0x7fec0811e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811dfe0 0x7fec0811e3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec0811f830 tx=0x7febf00046c0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811dfe0 msgr2=0x7fec0811e3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:23:14.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec04ff9700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811dfe0 0x7fec0811e3c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.736+0000 7fec07fff700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7febe80355e0 0x7febe8037aa0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fec081caf40 tx=0x7fec00003f00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:23:14.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811e900 0x7fec0806e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 1 -- 172.21.15.37:0/3775759286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0809e6c0 con 0x7fec0811e900 2024-09-16T07:23:14.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811e900 0x7fec0806e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 10 monclient: get_auth_request con 0x7fec0811e900 auth_method 0 2024-09-16T07:23:14.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 10 monclient(hunting): handle_auth_done global_id 44171 payload 824 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 1 -- 172.21.15.37:0/3775759286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fec08120580 con 0x7fec0811e900 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7febfeffd700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811e900 0x7fec0806e520 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febf0000c00 tx=0x7febf00047e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0027680 con 0x7fec0811e900 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 10 monclient: dump: 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.751 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4608 (secure 0 0 0) 0x7febf001e2e0 con 0x7fec0811e900 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811e900 msgr2=0x7fec0806e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811e900 0x7fec0806e520 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febf0000c00 tx=0x7febf00047e0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 1 -- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811e900 msgr2=0x7fec0806e520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:23:14.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec04ff9700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec0811e900 0x7fec0806e520 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7febf0000c00 tx=0x7febf00047e0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 26, 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-09-16T07:23:14.753 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.754 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:23:14.756 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:23:14.757 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.758 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.759 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:23:14.760 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:23:14.761 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:23:14.762 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.763 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:23:14.764 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:23:14.765 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.765 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.765 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:23:14.765 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:23:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 1 -- 172.21.15.37:0/3775759286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7febe80355e0 msgr2=0x7febe8037aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7febe80355e0 0x7febe8037aa0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fec081caf40 tx=0x7fec00003f00 comp rx=0 tx=0).stop 2024-09-16T07:23:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 10 monclient: shutdown 2024-09-16T07:23:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 1 -- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 msgr2=0x7fec081c9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 1 --2- 172.21.15.37:0/3775759286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec080a1530 0x7fec081c9e80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fec080a35a0 tx=0x7febf8004770 comp rx=0 tx=0).stop 2024-09-16T07:23:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec07fff700 1 -- 172.21.15.37:0/3775759286 reap_dead start 2024-09-16T07:23:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec07fff700 1 -- 172.21.15.37:0/3775759286 reap_dead start 2024-09-16T07:23:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 1 -- 172.21.15.37:0/3775759286 shutdown_connections 2024-09-16T07:23:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 1 -- 172.21.15.37:0/3775759286 >> 172.21.15.37:0/3775759286 conn(0x7fec080a61b0 msgr2=0x7fec08070e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 1 -- 172.21.15.37:0/3775759286 shutdown_connections 2024-09-16T07:23:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.744+0000 7fec0de81700 1 -- 172.21.15.37:0/3775759286 wait complete. 2024-09-16T07:23:14.767 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T07:23:14.767 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T07:23:14.768 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T07:23:14.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.876+0000 7f386ab73700 10 monclient: get_monmap_and_config 2024-09-16T07:23:14.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.876+0000 7f386ab73700 10 monclient: build_initial_monmap 2024-09-16T07:23:14.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 10 monclient: monmap: 2024-09-16T07:23:14.883 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:23:14.883 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.883 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:23:14.883437+0000 2024-09-16T07:23:14.883 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:23:14.883437+0000 2024-09-16T07:23:14.883 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:23:14.883 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.884 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:23:14.884 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:23:14.884 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:23:14.884 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:23:14.884 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:23:14.884 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 1 Processor -- start 2024-09-16T07:23:14.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 1 -- start start 2024-09-16T07:23:14.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 10 monclient: init 2024-09-16T07:23:14.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:14.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T07:23:14.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f386409aaa0 0x7f3864132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 10 monclient(hunting): picked mon.noname-c con 0x7f386409aaa0 addr v2:172.21.15.143:3300/0 2024-09-16T07:23:14.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f386409dbb0 addr v1:172.21.15.37:6789/0 2024-09-16T07:23:14.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f386409dfb0 addr v1:172.21.15.143:6789/0 2024-09-16T07:23:14.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386409e400 con 0x7f386409dbb0 2024-09-16T07:23:14.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386409ae80 con 0x7f386409dfb0 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3864073ca0 con 0x7f386409aaa0 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386ab73700 10 monclient(hunting): authenticate will time out at 1115.370944s 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869370700 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f386409dbb0 legacy=0x7f38640999e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f386409aaa0 0x7f3864132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386a372700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f386409dfb0 0x7f386409a0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.37:40938/0 (socket says 172.21.15.37:40938) 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f386a372700 1 -- 172.21.15.37:0/3027818914 learned_addr learned my addr 172.21.15.37:0/3027818914 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 10 monclient(hunting): get_auth_request con 0x7f386409aaa0 auth_method 0 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 10 monclient(hunting): handle_auth_done global_id 5791 payload 274 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 1 -- 172.21.15.37:0/3027818914 >> v1:172.21.15.143:6789/0 conn(0x7f386409dfb0 legacy=0x7f386409a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 1 -- 172.21.15.37:0/3027818914 >> v1:172.21.15.37:6789/0 conn(0x7f386409dbb0 legacy=0x7f38640999e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 10 monclient: _finish_hunting 0 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 1 monclient: found mon.noname-c 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.880+0000 7f3869b71700 1 -- 172.21.15.37:0/3027818914 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386409a800 con 0x7f386409aaa0 2024-09-16T07:23:14.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3869b71700 1 --2- 172.21.15.37:0/3027818914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386409aaa0 0x7f3864132140 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3860009850 tx=0x7f3860009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac511cdad1392640 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 1 -- 172.21.15.37:0/3027818914 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860004230 con 0x7f386409aaa0 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 10 monclient: dump: 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.891 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.892 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.892 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 10 monclient: _finish_auth 0 2024-09-16T07:23:14.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 10 monclient: _check_auth_tickets 2024-09-16T07:23:14.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:14.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 5 monclient: authenticate success, global_id 5791 2024-09-16T07:23:14.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 1 -- 172.21.15.37:0/3027818914 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3860004390 con 0x7f386409aaa0 2024-09-16T07:23:14.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:23:14.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:23:14.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 10 monclient: get_monmap_and_config success 2024-09-16T07:23:14.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 4 set_mon_vals no callback set 2024-09-16T07:23:14.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 1 -- 172.21.15.37:0/3027818914 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38600036c0 con 0x7f386409aaa0 2024-09-16T07:23:14.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:23:14.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f3868b6f700 10 monclient: dump: 2024-09-16T07:23:14.893 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.893 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.893 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.893 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.894 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.894 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.894 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.894 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.894 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.894 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 10 monclient: shutdown 2024-09-16T07:23:14.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:14.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 1 -- 172.21.15.37:0/3027818914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386409aaa0 msgr2=0x7f3864132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 1 --2- 172.21.15.37:0/3027818914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386409aaa0 0x7f3864132140 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3860009850 tx=0x7f3860009b60 comp rx=0 tx=0).stop 2024-09-16T07:23:14.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 1 -- 172.21.15.37:0/3027818914 shutdown_connections 2024-09-16T07:23:14.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 1 --2- 172.21.15.37:0/3027818914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386409aaa0 0x7f3864132140 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 1 -- 172.21.15.37:0/3027818914 >> 172.21.15.37:0/3027818914 conn(0x7f38640a3f60 msgr2=0x7f38640a4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 1 -- 172.21.15.37:0/3027818914 shutdown_connections 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 1 -- 172.21.15.37:0/3027818914 wait complete. 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 10 monclient: build_initial_monmap 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.884+0000 7f386ab73700 10 monclient: monmap: 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:23:14.902 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 1 Processor -- start 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 1 -- start start 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient: init 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient: _reopen_session rank -1 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f386409aaa0 0x7f38641c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient(hunting): picked mon.noname-a con 0x7f386409aaa0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f386409dbb0 0x7f38641c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient(hunting): picked mon.noname-c con 0x7f386409dbb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386409dfb0 0x7f386411f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient(hunting): picked mon.noname-b con 0x7f386409dfb0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38641219a0 con 0x7f386409aaa0 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3864121e80 con 0x7f386409dfb0 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3864120310 con 0x7f386409dbb0 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient(hunting): _renew_subs 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient(hunting): authenticate will time out at 1115.377635s 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869b71700 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f386409aaa0 msgr2=0x7f38641c80c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869b71700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f386409aaa0 0x7f38641c80c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:23:14.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f386409dbb0 0x7f38641c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f386409dbb0 0x7f38641c8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51522/0 (socket says 172.21.15.37:51522) 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 1 -- 172.21.15.37:0/2134430684 learned_addr learned my addr 172.21.15.37:0/2134430684 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 10 monclient(hunting): get_auth_request con 0x7f386409dbb0 auth_method 0 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386a372700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386409dfb0 0x7f386411f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386a372700 10 monclient(hunting): get_auth_request con 0x7f386409dfb0 auth_method 0 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386a372700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386a372700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386a372700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386a372700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386a372700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386a372700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 10 monclient(hunting): handle_auth_done global_id 44174 payload 824 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 1 -- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386409dfb0 msgr2=0x7f386411f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386409dfb0 0x7f386411f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 1 -- 172.21.15.37:0/2134430684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f386409aaa0 msgr2=0x7f38641c80c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f386409aaa0 0x7f38641c80c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 10 monclient: _finish_hunting 0 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 1 monclient: found mon.noname-c 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 1 -- 172.21.15.37:0/2134430684 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38600094b0 con 0x7f386409dbb0 2024-09-16T07:23:14.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f3869370700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f386409dbb0 0x7f38641c8600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f385400b7b0 tx=0x7f385400bac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854003e70 con 0x7f386409dbb0 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 10 monclient: dump: 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 10 monclient: _finish_auth 0 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 5 monclient: authenticate success, global_id 44174 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3854005070 con 0x7f386409dbb0 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient: _renew_subs 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854005420 con 0x7f386409dbb0 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385bfff700 4 set_mon_vals no callback set 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 1 -- 172.21.15.37:0/2134430684 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38641318a0 con 0x7f386409dbb0 2024-09-16T07:23:14.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f385a7fc700 10 monclient: dump: 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient: _renew_subs 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.888+0000 7f386ab73700 1 -- 172.21.15.37:0/2134430684 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38640a1250 con 0x7f386409dbb0 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f3854005580 con 0x7f386409dbb0 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f386ab73700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f386ab73700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386411f7d0 0x7f386411fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f386ab73700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f386ab73700 1 -- 172.21.15.37:0/2134430684 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38641306b0 con 0x7f386411f7d0 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386411f7d0 0x7f386411fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f38480355b0 0x7f3848037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 10 monclient: get_auth_request con 0x7f386411f7d0 auth_method 0 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 4738+0+0 (secure 0 0 0) 0x7f38540476a0 con 0x7f386409dbb0 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f386a372700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f38480355b0 0x7f3848037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f386a372700 10 monclient: get_auth_request con 0x7f38480355b0 auth_method 0 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 10 monclient(hunting): handle_auth_done global_id 5800 payload 824 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 1 -- 172.21.15.37:0/2134430684 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f386409c3a0 con 0x7f386411f7d0 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f3869b71700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386411f7d0 0x7f386411fbb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3860009fd0 tx=0x7f3860004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860003d60 con 0x7f386411f7d0 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f386a372700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f38480355b0 0x7f3848037a70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3864122650 tx=0x7f385c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:23:14.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 10 monclient: dump: 2024-09-16T07:23:14.912 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.912 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.912 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.912 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.912 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.912 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f386001f070 con 0x7f386411f7d0 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386411f7d0 msgr2=0x7f386411fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386411f7d0 0x7f386411fbb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3860009fd0 tx=0x7f3860004770 comp rx=0 tx=0).stop 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386411f7d0 msgr2=0x7f386411fbb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:23:14.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.892+0000 7f385a7fc700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f386411f7d0 0x7f386411fbb0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.904+0000 7f386ab73700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:23:14.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.904+0000 7f386ab73700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38640706e0 0x7f3864070af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.904+0000 7f386ab73700 10 monclient(hunting): start opening mon connection 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.904+0000 7f386ab73700 1 -- 172.21.15.37:0/2134430684 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3864067700 con 0x7f38640706e0 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.904+0000 7f3869b71700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38640706e0 0x7f3864070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.904+0000 7f3869b71700 10 monclient: get_auth_request con 0x7f38640706e0 auth_method 0 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f3869b71700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f3869b71700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f3869b71700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f3869b71700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f3869b71700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f3869b71700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f3869b71700 10 monclient(hunting): handle_auth_done global_id 5803 payload 824 2024-09-16T07:23:14.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f3869b71700 1 -- 172.21.15.37:0/2134430684 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3864122b00 con 0x7f38640706e0 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f3869b71700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38640706e0 0x7f3864070af0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3860000c00 tx=0x7f3860009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386002aa30 con 0x7f38640706e0 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 10 monclient: dump: 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:23:14.917 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:23:14.918 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:23:14.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4776 (secure 0 0 0) 0x7f386002ab90 con 0x7f38640706e0 2024-09-16T07:23:14.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:23:14.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:23:14.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38640706e0 msgr2=0x7f3864070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38640706e0 0x7f3864070af0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3860000c00 tx=0x7f3860009240 comp rx=0 tx=0).stop 2024-09-16T07:23:14.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 1 -- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38640706e0 msgr2=0x7f3864070af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:23:14.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f385a7fc700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38640706e0 0x7f3864070af0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.918 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:23:14.918 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 26, 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:23:14.919 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:23:14.920 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:23:14.921 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:23:14.922 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.923 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.926 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:23:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:23:14.929 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:23:14.930 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 8 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:23:14.931 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:23:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 1 -- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f38480355b0 msgr2=0x7f3848037a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f38480355b0 0x7f3848037a70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3864122650 tx=0x7f385c009040 comp rx=0 tx=0).stop 2024-09-16T07:23:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 10 monclient: shutdown 2024-09-16T07:23:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:23:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 1 -- 172.21.15.37:0/2134430684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f386409dbb0 msgr2=0x7f38641c8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:23:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f386409dbb0 0x7f38641c8600 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f385400b7b0 tx=0x7f385400bac0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386a372700 1 -- 172.21.15.37:0/2134430684 reap_dead start 2024-09-16T07:23:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 1 -- 172.21.15.37:0/2134430684 shutdown_connections 2024-09-16T07:23:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 1 --2- 172.21.15.37:0/2134430684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f386409dbb0 0x7f38641c8600 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:23:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 1 -- 172.21.15.37:0/2134430684 >> 172.21.15.37:0/2134430684 conn(0x7f38640a3f60 msgr2=0x7f386409f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:23:14.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 1 -- 172.21.15.37:0/2134430684 shutdown_connections 2024-09-16T07:23:14.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:23:14.908+0000 7f386ab73700 1 -- 172.21.15.37:0/2134430684 wait complete. 2024-09-16T07:23:14.934 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T07:23:14.934 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-16T07:23:15.892 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:15.892 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27363]: starting. 2024-09-16T07:23:15.892 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27363]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27326 2024-09-16T07:23:15.892 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27363]: created object 0... 2024-09-16T07:23:15.892 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27363]: created object 25... 2024-09-16T07:23:15.892 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27363]: created object 49... 2024-09-16T07:23:15.892 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27363]: finishing. 2024-09-16T07:23:15.892 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27363]: shutting down. 2024-09-16T07:23:16.332 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:16.332 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27365]: starting. 2024-09-16T07:23:16.332 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27365]: listing objects. 2024-09-16T07:23:16.332 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27365]: listed object 0... 2024-09-16T07:23:16.332 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27365]: listed object 25... 2024-09-16T07:23:16.332 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27365]: saw 50 objects 2024-09-16T07:23:16.332 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27365]: shutting down. 2024-09-16T07:23:16.752 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:16.752 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27364]: starting. 2024-09-16T07:23:16.752 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27364]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27326 2024-09-16T07:23:16.752 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27364]: shutting down. 2024-09-16T07:23:16.755 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:16.755 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:16.755 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:23:16.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:23:18.942 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27428]: starting. 2024-09-16T07:23:18.942 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27428]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27427 2024-09-16T07:23:18.942 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27428]: created object 0... 2024-09-16T07:23:18.942 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27428]: created object 25... 2024-09-16T07:23:18.942 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27428]: created object 49... 2024-09-16T07:23:18.942 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27428]: finishing. 2024-09-16T07:23:18.942 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27428]: shutting down. 2024-09-16T07:23:19.782 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27429]: starting. 2024-09-16T07:23:19.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27429]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27427 2024-09-16T07:23:19.783 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27429]: shutting down. 2024-09-16T07:23:21.810 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:21.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27464]: starting. 2024-09-16T07:23:21.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27464]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27427 2024-09-16T07:23:21.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27464]: created object 0... 2024-09-16T07:23:21.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27464]: created object 25... 2024-09-16T07:23:21.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27464]: created object 49... 2024-09-16T07:23:21.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27464]: finishing. 2024-09-16T07:23:21.811 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27464]: shutting down. 2024-09-16T07:23:21.843 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:21.843 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27466]: starting. 2024-09-16T07:23:21.843 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27466]: listing objects. 2024-09-16T07:23:21.843 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27466]: listed object 0... 2024-09-16T07:23:21.843 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27466]: listed object 25... 2024-09-16T07:23:21.844 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27466]: saw 50 objects 2024-09-16T07:23:21.844 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27466]: shutting down. 2024-09-16T07:23:22.663 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:22.663 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27465]: starting. 2024-09-16T07:23:22.663 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27465]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27427 2024-09-16T07:23:22.663 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27465]: shutting down. 2024-09-16T07:23:22.667 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:22.667 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:22.667 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:23:22.671 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:23:24.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27519]: starting. 2024-09-16T07:23:24.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27519]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27518 2024-09-16T07:23:24.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27519]: created object 0... 2024-09-16T07:23:24.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27519]: created object 25... 2024-09-16T07:23:24.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27519]: created object 49... 2024-09-16T07:23:24.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27519]: finishing. 2024-09-16T07:23:24.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27519]: shutting down. 2024-09-16T07:23:25.684 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27520]: starting. 2024-09-16T07:23:25.684 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27520]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27518 2024-09-16T07:23:25.684 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27520]: shutting down. 2024-09-16T07:23:27.864 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:27.864 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27555]: starting. 2024-09-16T07:23:27.864 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27555]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27518 2024-09-16T07:23:27.864 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27555]: created object 0... 2024-09-16T07:23:27.864 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27555]: created object 25... 2024-09-16T07:23:27.864 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27555]: created object 49... 2024-09-16T07:23:27.865 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27555]: finishing. 2024-09-16T07:23:27.865 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27555]: shutting down. 2024-09-16T07:23:27.893 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:27.893 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27557]: starting. 2024-09-16T07:23:27.893 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27557]: listing objects. 2024-09-16T07:23:27.893 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27557]: listed object 0... 2024-09-16T07:23:27.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27557]: listed object 25... 2024-09-16T07:23:27.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27557]: saw 50 objects 2024-09-16T07:23:27.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27557]: shutting down. 2024-09-16T07:23:28.722 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:28.722 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27556]: starting. 2024-09-16T07:23:28.722 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27556]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27518 2024-09-16T07:23:28.722 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27556]: shutting down. 2024-09-16T07:23:28.724 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:28.724 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:28.724 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:23:28.727 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:23:30.807 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27610]: starting. 2024-09-16T07:23:30.807 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27610]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27609 2024-09-16T07:23:30.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27610]: created object 0... 2024-09-16T07:23:30.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27610]: created object 25... 2024-09-16T07:23:30.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27610]: created object 49... 2024-09-16T07:23:30.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27610]: finishing. 2024-09-16T07:23:30.808 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27610]: shutting down. 2024-09-16T07:23:31.670 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27611]: starting. 2024-09-16T07:23:31.670 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27611]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27609 2024-09-16T07:23:31.670 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27611]: shutting down. 2024-09-16T07:23:33.823 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:33.823 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27646]: starting. 2024-09-16T07:23:33.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27646]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27609 2024-09-16T07:23:33.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27646]: created object 0... 2024-09-16T07:23:33.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27646]: created object 25... 2024-09-16T07:23:33.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27646]: created object 49... 2024-09-16T07:23:33.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27646]: finishing. 2024-09-16T07:23:33.824 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27646]: shutting down. 2024-09-16T07:23:33.853 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:33.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27648]: starting. 2024-09-16T07:23:33.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27648]: listing objects. 2024-09-16T07:23:33.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27648]: listed object 0... 2024-09-16T07:23:33.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27648]: listed object 25... 2024-09-16T07:23:33.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27648]: saw 50 objects 2024-09-16T07:23:33.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27648]: shutting down. 2024-09-16T07:23:34.697 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:34.697 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27647]: starting. 2024-09-16T07:23:34.697 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27647]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27609 2024-09-16T07:23:34.697 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27647]: shutting down. 2024-09-16T07:23:34.701 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:34.701 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:34.701 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:23:34.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:23:36.861 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27701]: starting. 2024-09-16T07:23:36.862 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27701]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27700 2024-09-16T07:23:36.862 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27701]: created object 0... 2024-09-16T07:23:36.862 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27701]: created object 25... 2024-09-16T07:23:36.862 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27701]: created object 49... 2024-09-16T07:23:36.862 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27701]: finishing. 2024-09-16T07:23:36.862 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27701]: shutting down. 2024-09-16T07:23:37.716 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27702]: starting. 2024-09-16T07:23:37.717 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27702]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27700 2024-09-16T07:23:37.717 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27702]: shutting down. 2024-09-16T07:23:39.901 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:39.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27737]: starting. 2024-09-16T07:23:39.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27737]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27700 2024-09-16T07:23:39.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27737]: created object 0... 2024-09-16T07:23:39.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27737]: created object 25... 2024-09-16T07:23:39.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27737]: created object 49... 2024-09-16T07:23:39.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27737]: finishing. 2024-09-16T07:23:39.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27737]: shutting down. 2024-09-16T07:23:39.932 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:39.933 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27739]: starting. 2024-09-16T07:23:39.933 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27739]: listing objects. 2024-09-16T07:23:39.933 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27739]: listed object 0... 2024-09-16T07:23:39.933 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27739]: listed object 25... 2024-09-16T07:23:39.933 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27739]: saw 50 objects 2024-09-16T07:23:39.933 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27739]: shutting down. 2024-09-16T07:23:40.661 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:40.661 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27738]: starting. 2024-09-16T07:23:40.662 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27738]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27700 2024-09-16T07:23:40.662 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27738]: shutting down. 2024-09-16T07:23:40.664 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:40.664 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:40.665 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:23:40.667 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:23:42.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27792]: starting. 2024-09-16T07:23:42.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27792]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27791 2024-09-16T07:23:42.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27792]: created object 0... 2024-09-16T07:23:42.841 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27792]: created object 25... 2024-09-16T07:23:42.842 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27792]: created object 49... 2024-09-16T07:23:42.842 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27792]: finishing. 2024-09-16T07:23:42.842 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27792]: shutting down. 2024-09-16T07:23:43.682 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27793]: starting. 2024-09-16T07:23:43.682 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27793]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27791 2024-09-16T07:23:43.682 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27793]: shutting down. 2024-09-16T07:23:45.852 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:45.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27828]: starting. 2024-09-16T07:23:45.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27828]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27791 2024-09-16T07:23:45.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27828]: created object 0... 2024-09-16T07:23:45.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27828]: created object 25... 2024-09-16T07:23:45.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27828]: created object 49... 2024-09-16T07:23:45.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27828]: finishing. 2024-09-16T07:23:45.854 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27828]: shutting down. 2024-09-16T07:23:45.881 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:45.881 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27830]: starting. 2024-09-16T07:23:45.881 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27830]: listing objects. 2024-09-16T07:23:45.881 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27830]: listed object 0... 2024-09-16T07:23:45.881 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27830]: listed object 25... 2024-09-16T07:23:45.881 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27830]: saw 50 objects 2024-09-16T07:23:45.881 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27830]: shutting down. 2024-09-16T07:23:46.725 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:46.725 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27829]: starting. 2024-09-16T07:23:46.725 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27829]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27791 2024-09-16T07:23:46.725 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27829]: shutting down. 2024-09-16T07:23:46.729 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:46.729 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:46.729 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:23:46.733 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:23:48.885 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27883]: starting. 2024-09-16T07:23:48.885 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27883]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27882 2024-09-16T07:23:48.886 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27883]: created object 0... 2024-09-16T07:23:48.886 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27883]: created object 25... 2024-09-16T07:23:48.886 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27883]: created object 49... 2024-09-16T07:23:48.886 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27883]: finishing. 2024-09-16T07:23:48.886 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27883]: shutting down. 2024-09-16T07:23:49.751 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27884]: starting. 2024-09-16T07:23:49.751 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27884]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27882 2024-09-16T07:23:49.751 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27884]: shutting down. 2024-09-16T07:23:51.822 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:51.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27919]: starting. 2024-09-16T07:23:51.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27919]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27882 2024-09-16T07:23:51.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27919]: created object 0... 2024-09-16T07:23:51.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27919]: created object 25... 2024-09-16T07:23:51.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27919]: created object 49... 2024-09-16T07:23:51.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27919]: finishing. 2024-09-16T07:23:51.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[27919]: shutting down. 2024-09-16T07:23:51.853 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:51.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27921]: starting. 2024-09-16T07:23:51.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27921]: listing objects. 2024-09-16T07:23:51.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27921]: listed object 0... 2024-09-16T07:23:51.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27921]: listed object 25... 2024-09-16T07:23:51.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27921]: saw 50 objects 2024-09-16T07:23:51.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[27921]: shutting down. 2024-09-16T07:23:52.680 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:52.680 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27920]: starting. 2024-09-16T07:23:52.680 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27920]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27882 2024-09-16T07:23:52.681 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[27920]: shutting down. 2024-09-16T07:23:52.684 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:52.684 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:52.684 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:23:52.688 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:23:54.840 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27974]: starting. 2024-09-16T07:23:55.053 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27974]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27973 2024-09-16T07:23:55.053 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27974]: created object 0... 2024-09-16T07:23:55.053 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27974]: created object 25... 2024-09-16T07:23:55.053 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27974]: created object 49... 2024-09-16T07:23:55.053 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27974]: finishing. 2024-09-16T07:23:55.053 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[27974]: shutting down. 2024-09-16T07:23:55.697 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27975]: starting. 2024-09-16T07:23:55.697 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27975]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27973 2024-09-16T07:23:55.697 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[27975]: shutting down. 2024-09-16T07:23:57.874 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:57.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28010]: starting. 2024-09-16T07:23:57.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28010]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-27973 2024-09-16T07:23:57.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28010]: created object 0... 2024-09-16T07:23:57.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28010]: created object 25... 2024-09-16T07:23:57.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28010]: created object 49... 2024-09-16T07:23:57.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28010]: finishing. 2024-09-16T07:23:57.875 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28010]: shutting down. 2024-09-16T07:23:57.905 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:57.908 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28012]: starting. 2024-09-16T07:23:57.908 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28012]: listing objects. 2024-09-16T07:23:57.908 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28012]: listed object 0... 2024-09-16T07:23:57.909 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28012]: listed object 25... 2024-09-16T07:23:57.909 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28012]: saw 50 objects 2024-09-16T07:23:57.909 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28012]: shutting down. 2024-09-16T07:23:58.733 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:58.734 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28011]: starting. 2024-09-16T07:23:58.734 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28011]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-27973 2024-09-16T07:23:58.734 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28011]: shutting down. 2024-09-16T07:23:58.736 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:58.737 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:23:58.737 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:23:58.740 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:24:00.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28065]: starting. 2024-09-16T07:24:00.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28065]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28064 2024-09-16T07:24:00.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28065]: created object 0... 2024-09-16T07:24:00.816 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28065]: created object 25... 2024-09-16T07:24:00.817 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28065]: created object 49... 2024-09-16T07:24:00.817 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28065]: finishing. 2024-09-16T07:24:00.817 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28065]: shutting down. 2024-09-16T07:24:01.671 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28066]: starting. 2024-09-16T07:24:01.672 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28066]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28064 2024-09-16T07:24:01.672 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28066]: shutting down. 2024-09-16T07:24:03.833 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:03.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28101]: starting. 2024-09-16T07:24:03.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28101]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28064 2024-09-16T07:24:03.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28101]: created object 0... 2024-09-16T07:24:03.834 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28101]: created object 25... 2024-09-16T07:24:03.834 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28101]: created object 49... 2024-09-16T07:24:03.834 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28101]: finishing. 2024-09-16T07:24:03.834 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28101]: shutting down. 2024-09-16T07:24:03.862 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:03.862 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28103]: starting. 2024-09-16T07:24:03.862 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28103]: listing objects. 2024-09-16T07:24:03.862 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28103]: listed object 0... 2024-09-16T07:24:03.863 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28103]: listed object 25... 2024-09-16T07:24:03.863 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28103]: saw 50 objects 2024-09-16T07:24:03.863 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28103]: shutting down. 2024-09-16T07:24:04.699 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:04.699 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28102]: starting. 2024-09-16T07:24:04.700 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28102]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28064 2024-09-16T07:24:04.700 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28102]: shutting down. 2024-09-16T07:24:04.702 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:04.703 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:04.703 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:24:04.706 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:24:07.078 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28156]: starting. 2024-09-16T07:24:07.079 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28156]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28155 2024-09-16T07:24:07.079 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28156]: created object 0... 2024-09-16T07:24:07.079 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28156]: created object 25... 2024-09-16T07:24:07.079 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28156]: created object 49... 2024-09-16T07:24:07.079 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28156]: finishing. 2024-09-16T07:24:07.079 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28156]: shutting down. 2024-09-16T07:24:07.709 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28157]: starting. 2024-09-16T07:24:07.709 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28157]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28155 2024-09-16T07:24:07.709 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28157]: shutting down. 2024-09-16T07:24:09.898 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:09.898 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28192]: starting. 2024-09-16T07:24:09.898 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28192]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28155 2024-09-16T07:24:09.898 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28192]: created object 0... 2024-09-16T07:24:09.898 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28192]: created object 25... 2024-09-16T07:24:09.899 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28192]: created object 49... 2024-09-16T07:24:09.899 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28192]: finishing. 2024-09-16T07:24:09.899 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28192]: shutting down. 2024-09-16T07:24:09.928 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:09.928 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28194]: starting. 2024-09-16T07:24:09.928 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28194]: listing objects. 2024-09-16T07:24:09.928 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28194]: listed object 0... 2024-09-16T07:24:09.929 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28194]: listed object 25... 2024-09-16T07:24:09.929 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28194]: saw 50 objects 2024-09-16T07:24:09.929 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28194]: shutting down. 2024-09-16T07:24:10.674 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:10.674 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28193]: starting. 2024-09-16T07:24:10.674 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28193]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28155 2024-09-16T07:24:10.674 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28193]: shutting down. 2024-09-16T07:24:10.677 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:10.677 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:10.677 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:24:10.680 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:24:12.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28247]: starting. 2024-09-16T07:24:12.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28247]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28246 2024-09-16T07:24:12.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28247]: created object 0... 2024-09-16T07:24:12.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28247]: created object 25... 2024-09-16T07:24:12.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28247]: created object 49... 2024-09-16T07:24:12.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28247]: finishing. 2024-09-16T07:24:12.833 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28247]: shutting down. 2024-09-16T07:24:13.691 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28248]: starting. 2024-09-16T07:24:13.692 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28248]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28246 2024-09-16T07:24:13.692 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28248]: shutting down. 2024-09-16T07:24:15.873 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:15.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28284]: starting. 2024-09-16T07:24:15.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28284]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28246 2024-09-16T07:24:15.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28284]: created object 0... 2024-09-16T07:24:15.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28284]: created object 25... 2024-09-16T07:24:15.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28284]: created object 49... 2024-09-16T07:24:15.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28284]: finishing. 2024-09-16T07:24:15.874 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28284]: shutting down. 2024-09-16T07:24:15.907 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:15.907 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28286]: starting. 2024-09-16T07:24:15.907 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28286]: listing objects. 2024-09-16T07:24:15.907 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28286]: listed object 0... 2024-09-16T07:24:15.907 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28286]: listed object 25... 2024-09-16T07:24:15.908 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28286]: saw 50 objects 2024-09-16T07:24:15.908 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28286]: shutting down. 2024-09-16T07:24:16.731 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:16.731 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28285]: starting. 2024-09-16T07:24:16.731 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28285]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28246 2024-09-16T07:24:16.731 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28285]: shutting down. 2024-09-16T07:24:16.734 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:16.734 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:16.734 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:24:16.737 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:24:18.893 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28339]: starting. 2024-09-16T07:24:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28339]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28338 2024-09-16T07:24:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28339]: created object 0... 2024-09-16T07:24:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28339]: created object 25... 2024-09-16T07:24:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28339]: created object 49... 2024-09-16T07:24:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28339]: finishing. 2024-09-16T07:24:18.894 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28339]: shutting down. 2024-09-16T07:24:19.757 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28340]: starting. 2024-09-16T07:24:19.757 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28340]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28338 2024-09-16T07:24:19.757 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28340]: shutting down. 2024-09-16T07:24:21.821 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:21.821 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28375]: starting. 2024-09-16T07:24:21.821 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28375]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28338 2024-09-16T07:24:21.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28375]: created object 0... 2024-09-16T07:24:21.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28375]: created object 25... 2024-09-16T07:24:21.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28375]: created object 49... 2024-09-16T07:24:21.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28375]: finishing. 2024-09-16T07:24:21.822 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28375]: shutting down. 2024-09-16T07:24:21.852 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:21.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28377]: starting. 2024-09-16T07:24:21.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28377]: listing objects. 2024-09-16T07:24:21.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28377]: listed object 0... 2024-09-16T07:24:21.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28377]: listed object 25... 2024-09-16T07:24:21.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28377]: saw 50 objects 2024-09-16T07:24:21.852 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28377]: shutting down. 2024-09-16T07:24:22.678 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:22.678 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28376]: starting. 2024-09-16T07:24:22.678 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28376]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28338 2024-09-16T07:24:22.678 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28376]: shutting down. 2024-09-16T07:24:22.682 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:22.682 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:22.682 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:24:22.686 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:24:24.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28430]: starting. 2024-09-16T07:24:24.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28430]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28429 2024-09-16T07:24:24.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28430]: created object 0... 2024-09-16T07:24:24.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28430]: created object 25... 2024-09-16T07:24:24.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28430]: created object 49... 2024-09-16T07:24:24.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28430]: finishing. 2024-09-16T07:24:24.853 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28430]: shutting down. 2024-09-16T07:24:25.689 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28431]: starting. 2024-09-16T07:24:25.689 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28431]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28429 2024-09-16T07:24:25.689 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28431]: shutting down. 2024-09-16T07:24:27.880 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:27.880 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28466]: starting. 2024-09-16T07:24:27.880 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28466]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28429 2024-09-16T07:24:27.880 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28466]: created object 0... 2024-09-16T07:24:27.880 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28466]: created object 25... 2024-09-16T07:24:27.880 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28466]: created object 49... 2024-09-16T07:24:27.881 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28466]: finishing. 2024-09-16T07:24:27.881 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28466]: shutting down. 2024-09-16T07:24:27.911 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:27.911 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28468]: starting. 2024-09-16T07:24:27.911 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28468]: listing objects. 2024-09-16T07:24:27.911 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28468]: listed object 0... 2024-09-16T07:24:27.911 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28468]: listed object 25... 2024-09-16T07:24:27.911 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28468]: saw 50 objects 2024-09-16T07:24:27.911 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28468]: shutting down. 2024-09-16T07:24:28.735 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:28.735 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28467]: starting. 2024-09-16T07:24:28.735 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28467]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28429 2024-09-16T07:24:28.735 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28467]: shutting down. 2024-09-16T07:24:28.738 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:28.738 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:28.738 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:24:28.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:24:30.828 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28521]: starting. 2024-09-16T07:24:30.829 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28521]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28520 2024-09-16T07:24:30.829 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28521]: created object 0... 2024-09-16T07:24:30.829 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28521]: created object 25... 2024-09-16T07:24:30.829 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28521]: created object 49... 2024-09-16T07:24:30.829 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28521]: finishing. 2024-09-16T07:24:30.829 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28521]: shutting down. 2024-09-16T07:24:31.693 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28522]: starting. 2024-09-16T07:24:31.693 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28522]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28520 2024-09-16T07:24:31.693 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28522]: shutting down. 2024-09-16T07:24:33.857 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:33.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28557]: starting. 2024-09-16T07:24:33.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28557]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28520 2024-09-16T07:24:33.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28557]: created object 0... 2024-09-16T07:24:33.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28557]: created object 25... 2024-09-16T07:24:33.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28557]: created object 49... 2024-09-16T07:24:33.857 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28557]: finishing. 2024-09-16T07:24:33.858 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28557]: shutting down. 2024-09-16T07:24:33.889 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:33.889 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28559]: starting. 2024-09-16T07:24:33.889 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28559]: listing objects. 2024-09-16T07:24:33.889 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28559]: listed object 0... 2024-09-16T07:24:33.890 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28559]: listed object 25... 2024-09-16T07:24:33.890 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28559]: saw 50 objects 2024-09-16T07:24:33.890 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28559]: shutting down. 2024-09-16T07:24:34.717 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:34.717 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28558]: starting. 2024-09-16T07:24:34.717 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28558]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28520 2024-09-16T07:24:34.717 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28558]: shutting down. 2024-09-16T07:24:34.720 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:34.721 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:34.721 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:24:34.723 DEBUG:teuthology.orchestra.run.smithi143:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-16T07:24:36.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28612]: starting. 2024-09-16T07:24:36.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28612]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28611 2024-09-16T07:24:36.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28612]: created object 0... 2024-09-16T07:24:36.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28612]: created object 25... 2024-09-16T07:24:36.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28612]: created object 49... 2024-09-16T07:24:36.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28612]: finishing. 2024-09-16T07:24:36.901 INFO:teuthology.orchestra.run.smithi143.stdout:process_1_[28612]: shutting down. 2024-09-16T07:24:37.742 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28613]: starting. 2024-09-16T07:24:37.742 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28613]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28611 2024-09-16T07:24:37.742 INFO:teuthology.orchestra.run.smithi143.stdout:process_2_[28613]: shutting down. 2024-09-16T07:24:39.920 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:39.921 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28648]: starting. 2024-09-16T07:24:39.921 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28648]: creating pool ceph_test_rados_delete_pools_parallel.smithi143-28611 2024-09-16T07:24:39.921 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28648]: created object 0... 2024-09-16T07:24:39.921 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28648]: created object 25... 2024-09-16T07:24:39.921 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28648]: created object 49... 2024-09-16T07:24:39.921 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28648]: finishing. 2024-09-16T07:24:39.921 INFO:teuthology.orchestra.run.smithi143.stdout:process_3_[28648]: shutting down. 2024-09-16T07:24:39.952 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:39.952 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28650]: starting. 2024-09-16T07:24:39.952 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28650]: listing objects. 2024-09-16T07:24:39.952 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28650]: listed object 0... 2024-09-16T07:24:39.952 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28650]: listed object 25... 2024-09-16T07:24:39.952 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28650]: saw 50 objects 2024-09-16T07:24:39.952 INFO:teuthology.orchestra.run.smithi143.stdout:process_5_[28650]: shutting down. 2024-09-16T07:24:40.682 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:40.682 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28649]: starting. 2024-09-16T07:24:40.682 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28649]: deleting pool ceph_test_rados_delete_pools_parallel.smithi143-28611 2024-09-16T07:24:40.682 INFO:teuthology.orchestra.run.smithi143.stdout:process_4_[28649]: shutting down. 2024-09-16T07:24:40.685 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:40.685 INFO:teuthology.orchestra.run.smithi143.stdout:******************************* 2024-09-16T07:24:40.685 INFO:teuthology.orchestra.run.smithi143.stdout:******* SUCCESS ********** 2024-09-16T07:24:40.688 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T07:24:40.697 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T07:24:44.935 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T07:24:44.936 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T07:24:44.936 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T07:24:44.936 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T07:24:44.939 INFO:tasks.ceph.mon.a:Started 2024-09-16T07:24:44.939 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T07:24:44.939 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:24:45.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient: get_monmap_and_config 2024-09-16T07:24:45.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient: build_initial_monmap 2024-09-16T07:24:45.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient: monmap: 2024-09-16T07:24:45.056 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:45.056 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:45.056 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:24:45.055606+0000 2024-09-16T07:24:45.056 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:24:45.055606+0000 2024-09-16T07:24:45.056 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:45.057 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:45.057 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:24:45.057 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:24:45.057 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:24:45.057 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:24:45.057 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:24:45.057 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:45.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 1 Processor -- start 2024-09-16T07:24:45.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 1 -- start start 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient: init 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fae6c0b87a0 0x7fae6c024540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient(hunting): picked mon.noname-a con 0x7fae6c0b87a0 addr v2:172.21.15.37:3300/0 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae6c0b8b80 addr v1:172.21.15.37:6789/0 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8f90 0x7fae6c006db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient(hunting): picked mon.noname-b con 0x7fae6c0b8f90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae6c023880 con 0x7fae6c0b8b80 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae6c023610 con 0x7fae6c0b87a0 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:45.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c0226e0 con 0x7fae6c0b8f90 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient(hunting): authenticate will time out at 1205.542097s 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae79199700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fae6c0b8b80 0x7fae6c0066a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53092/0 (socket says 172.21.15.37:53092) 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae79199700 1 -- 172.21.15.37:0/3680316359 learned_addr learned my addr 172.21.15.37:0/3680316359 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 1 --2- 172.21.15.37:0/3680316359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8f90 0x7fae6c006db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7999a700 1 --2- 172.21.15.37:0/3680316359 >> v2:172.21.15.37:3300/0 conn(0x7fae6c0b87a0 0x7fae6c024540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 10 monclient(hunting): get_auth_request con 0x7fae6c0b8f90 auth_method 0 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7999a700 10 monclient(hunting): get_auth_request con 0x7fae6c0b87a0 auth_method 0 2024-09-16T07:24:45.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7999a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7999a700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7999a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7999a700 1 -- 172.21.15.37:0/3680316359 >> v2:172.21.15.37:3300/0 conn(0x7fae6c0b87a0 msgr2=0x7fae6c024540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7999a700 1 -- 172.21.15.37:0/3680316359 >> v2:172.21.15.37:3300/0 conn(0x7fae6c0b87a0 msgr2=0x7fae6c024540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 10 monclient(hunting): handle_auth_done global_id 44483 payload 274 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7999a700 1 --2- 172.21.15.37:0/3680316359 >> v2:172.21.15.37:3300/0 conn(0x7fae6c0b87a0 0x7fae6c024540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 1 -- 172.21.15.37:0/3680316359 >> v2:172.21.15.37:3300/0 conn(0x7fae6c0b87a0 msgr2=0x7fae6c024540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7999a700 1 --2- 172.21.15.37:0/3680316359 >> v2:172.21.15.37:3300/0 conn(0x7fae6c0b87a0 0x7fae6c024540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 1 --2- 172.21.15.37:0/3680316359 >> v2:172.21.15.37:3300/0 conn(0x7fae6c0b87a0 0x7fae6c024540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:45.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 1 -- 172.21.15.37:0/3680316359 >> v1:172.21.15.37:6789/0 conn(0x7fae6c0b8b80 legacy=0x7fae6c0066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 10 monclient: _finish_hunting 0 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 1 monclient: found mon.noname-b 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 1 -- 172.21.15.37:0/3680316359 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6c022220 con 0x7fae6c0b8f90 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a19b700 1 --2- 172.21.15.37:0/3680316359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8f90 0x7fae6c006db0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fae74068a10 tx=0x7fae74066f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d82c9ea7521bd35 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 1 -- 172.21.15.37:0/3680316359 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7406a310 con 0x7fae6c0b8f90 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 10 monclient: dump: 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:45.069 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 10 monclient: _finish_auth 0 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 10 monclient: _check_auth_tickets 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 5 monclient: authenticate success, global_id 44483 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 1 -- 172.21.15.37:0/3680316359 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7408e070 con 0x7fae6c0b8f90 2024-09-16T07:24:45.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 10 monclient: get_monmap_and_config success 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 1 -- 172.21.15.37:0/3680316359 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74067280 con 0x7fae6c0b8f90 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae7a99c700 4 set_mon_vals no callback set 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.054+0000 7fae78998700 10 monclient: dump: 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:45.071 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient: shutdown 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 -- 172.21.15.37:0/3680316359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8f90 msgr2=0x7fae6c006db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 --2- 172.21.15.37:0/3680316359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8f90 0x7fae6c006db0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fae74068a10 tx=0x7fae74066f10 comp rx=0 tx=0).stop 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 -- 172.21.15.37:0/3680316359 shutdown_connections 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 --2- 172.21.15.37:0/3680316359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8f90 0x7fae6c006db0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 --2- 172.21.15.37:0/3680316359 >> v2:172.21.15.37:3300/0 conn(0x7fae6c0b87a0 0x7fae6c024540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 -- 172.21.15.37:0/3680316359 >> 172.21.15.37:0/3680316359 conn(0x7fae6c027700 msgr2=0x7fae6c027b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 -- 172.21.15.37:0/3680316359 shutdown_connections 2024-09-16T07:24:45.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 -- 172.21.15.37:0/3680316359 wait complete. 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient: build_initial_monmap 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient: monmap: 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:45.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 Processor -- start 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 -- start start 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient: init 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae6c0b87a0 0x7fae6c150510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient(hunting): picked mon.noname-b con 0x7fae6c0b87a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8b80 0x7fae6c0b7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient(hunting): picked mon.noname-c con 0x7fae6c0b8b80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae6c0b8f90 0x7fae6c0b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient(hunting): picked mon.noname-a con 0x7fae6c0b8f90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c0a9e00 con 0x7fae6c0b8f90 2024-09-16T07:24:45.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c150a50 con 0x7fae6c0b87a0 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6c0abe00 con 0x7fae6c0b8b80 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a99c700 10 monclient(hunting): authenticate will time out at 1205.546742s 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8b80 0x7fae6c0b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8b80 0x7fae6c0b7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40752/0 (socket says 172.21.15.37:40752) 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a19b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae6c0b8f90 0x7fae6c0b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 1 -- 172.21.15.37:0/1954613926 learned_addr learned my addr 172.21.15.37:0/1954613926 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 10 monclient(hunting): get_auth_request con 0x7fae6c0b8b80 auth_method 0 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:45.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a19b700 10 monclient(hunting): get_auth_request con 0x7fae6c0b8f90 auth_method 0 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a19b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a19b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a19b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7999a700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae6c0b87a0 0x7fae6c150510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7999a700 10 monclient(hunting): get_auth_request con 0x7fae6c0b87a0 auth_method 0 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7999a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7999a700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7999a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a19b700 1 -- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae6c0b8f90 msgr2=0x7fae6c0b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:24:45.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a19b700 1 -- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae6c0b8f90 msgr2=0x7fae6c0b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a19b700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae6c0b8f90 0x7fae6c0b7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7a19b700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae6c0b8f90 0x7fae6c0b7b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 10 monclient(hunting): handle_auth_done global_id 44486 payload 824 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 1 -- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae6c0b87a0 msgr2=0x7fae6c150510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae6c0b87a0 0x7fae6c150510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 1 -- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae6c0b8f90 msgr2=0x7fae6c0b7b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae6c0b8f90 0x7fae6c0b7b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 10 monclient: _finish_hunting 0 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 1 monclient: found mon.noname-c 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:24:45.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 1 -- 172.21.15.37:0/1954613926 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7404f5c0 con 0x7fae6c0b8b80 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae7999a700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae6c0b87a0 0x7fae6c150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae79199700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8b80 0x7fae6c0b7620 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fae6400c700 tx=0x7fae6400cac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.058+0000 7fae6a7fc700 1 -- 172.21.15.37:0/1954613926 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae64005d90 con 0x7fae6c0b8b80 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 10 monclient: dump: 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:45.078 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 10 monclient: _finish_auth 0 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 5 monclient: authenticate success, global_id 44486 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 1 -- 172.21.15.37:0/1954613926 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae640070d0 con 0x7fae6c0b8b80 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 1 -- 172.21.15.37:0/1954613926 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae64007440 con 0x7fae6c0b8b80 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 10 monclient: _renew_subs 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:45.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 1 -- 172.21.15.37:0/1954613926 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6c0219b0 con 0x7fae6c0b8b80 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6bfff700 4 set_mon_vals no callback set 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 10 monclient: dump: 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:45.080 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 10 monclient: _renew_subs 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 1 -- 172.21.15.37:0/1954613926 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6c02a190 con 0x7fae6c0b8b80 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7a99c700 1 -- 172.21.15.37:0/1954613926 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6c024540 con 0x7fae6c0b8b80 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 1 -- 172.21.15.37:0/1954613926 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fae640075a0 con 0x7fae6c0b8b80 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fae58035600 0x7fae58037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:45.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae6a7fc700 1 -- 172.21.15.37:0/1954613926 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fae64046f10 con 0x7fae6c0b8b80 2024-09-16T07:24:45.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7999a700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fae58035600 0x7fae58037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:45.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7999a700 10 monclient: get_auth_request con 0x7fae58035600 auth_method 0 2024-09-16T07:24:45.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.062+0000 7fae7999a700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fae58035600 0x7fae58037ac0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fae70009100 tx=0x7fae70008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:45.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.070+0000 7fae6a7fc700 1 -- 172.21.15.37:0/1954613926 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fae64019b40 con 0x7fae6c0b8b80 2024-09-16T07:24:45.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.070+0000 7fae6a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:45.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.070+0000 7fae6a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:24:45.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:24:45.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:24:45.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:45.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 -- 172.21.15.37:0/1954613926 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fae6c0ab870 con 0x7fae6c0b8b80 2024-09-16T07:24:45.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae6a7fc700 1 -- 172.21.15.37:0/1954613926 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7fae64017070 con 0x7fae6c0b8b80 2024-09-16T07:24:45.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae6a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:24:45.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae6a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:24:45.305 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":91,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:24:45.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 -- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fae58035600 msgr2=0x7fae58037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:45.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fae58035600 0x7fae58037ac0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fae70009100 tx=0x7fae70008040 comp rx=0 tx=0).stop 2024-09-16T07:24:45.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 10 monclient: shutdown 2024-09-16T07:24:45.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:45.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 -- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8b80 msgr2=0x7fae6c0b7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:45.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8b80 0x7fae6c0b7620 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fae6400c700 tx=0x7fae6400cac0 comp rx=0 tx=0).stop 2024-09-16T07:24:45.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 -- 172.21.15.37:0/1954613926 shutdown_connections 2024-09-16T07:24:45.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fae6c0b8f90 0x7fae6c0b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:45.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fae58035600 0x7fae58037ac0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:45.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fae6c0b8b80 0x7fae6c0b7620 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:45.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 --2- 172.21.15.37:0/1954613926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae6c0b87a0 0x7fae6c150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:45.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 -- 172.21.15.37:0/1954613926 >> 172.21.15.37:0/1954613926 conn(0x7fae6c027700 msgr2=0x7fae6c01fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:45.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 -- 172.21.15.37:0/1954613926 shutdown_connections 2024-09-16T07:24:45.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:45.302+0000 7fae7a99c700 1 -- 172.21.15.37:0/1954613926 wait complete. 2024-09-16T07:24:48.321 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:24:48.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient: get_monmap_and_config 2024-09-16T07:24:48.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient: build_initial_monmap 2024-09-16T07:24:48.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient: monmap: 2024-09-16T07:24:48.444 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:48.445 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:48.445 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:24:48.443940+0000 2024-09-16T07:24:48.445 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:24:48.443940+0000 2024-09-16T07:24:48.445 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:48.445 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:48.445 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:24:48.445 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:24:48.445 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:24:48.445 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:24:48.445 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:24:48.446 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:48.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 1 Processor -- start 2024-09-16T07:24:48.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 1 -- start start 2024-09-16T07:24:48.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient: init 2024-09-16T07:24:48.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:48.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T07:24:48.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1edc09a1c0 addr v1:172.21.15.143:6789/0 2024-09-16T07:24:48.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1edc06b600 addr v1:172.21.15.37:6789/0 2024-09-16T07:24:48.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1edc0a0a80 0x7f1edc13acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:48.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f1edc0a0a80 addr v2:172.21.15.143:3300/0 2024-09-16T07:24:48.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc09a640 con 0x7f1edc06b600 2024-09-16T07:24:48.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc0a0e90 con 0x7f1edc09a1c0 2024-09-16T07:24:48.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:48.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1edc09bda0 con 0x7f1edc0a0a80 2024-09-16T07:24:48.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:48.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edbfff700 10 monclient(hunting): authenticate will time out at 1208.929878s 2024-09-16T07:24:48.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1eda7fc700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1edc06b600 0x7f1edc1375a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53104/0 (socket says 172.21.15.37:53104) 2024-09-16T07:24:48.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1eda7fc700 1 -- 172.21.15.37:0/3957866306 learned_addr learned my addr 172.21.15.37:0/3957866306 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:24:48.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 1 --2- 172.21.15.37:0/3957866306 >> v2:172.21.15.143:3300/0 conn(0x7f1edc0a0a80 0x7f1edc13acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:48.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 10 monclient(hunting): get_auth_request con 0x7f1edc0a0a80 auth_method 0 2024-09-16T07:24:48.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:48.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:48.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:48.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:48.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1ed9ffb700 1 -- 172.21.15.37:0/3957866306 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524211770 0 0) 0x7f1edc0a0e90 con 0x7f1edc09a1c0 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1ed9ffb700 10 monclient(hunting): _negotiate 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1ed9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1ed9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1ed9ffb700 10 monclient(hunting): my global_id is 6253 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1ed9ffb700 1 -- 172.21.15.37:0/3957866306 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec80035d0 con 0x7f1edc09a1c0 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 10 monclient(hunting): handle_auth_done global_id 6250 payload 274 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 1 -- 172.21.15.37:0/3957866306 >> v1:172.21.15.143:6789/0 conn(0x7f1edc09a1c0 legacy=0x7f1edc09d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 1 -- 172.21.15.37:0/3957866306 >> v1:172.21.15.37:6789/0 conn(0x7f1edc06b600 legacy=0x7f1edc1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:48.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 1 monclient: found mon.noname-c 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 1 -- 172.21.15.37:0/3957866306 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc131c30 con 0x7f1edc0a0a80 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.442+0000 7f1edb7fe700 1 --2- 172.21.15.37:0/3957866306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1edc0a0a80 0x7f1edc13acd0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4009850 tx=0x7f1ed4009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f859a267a81ab35 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 1 -- 172.21.15.37:0/3957866306 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4011830 con 0x7f1edc0a0a80 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 10 monclient: dump: 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:48.455 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 10 monclient: _finish_auth 0 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 5 monclient: authenticate success, global_id 6250 2024-09-16T07:24:48.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 1 -- 172.21.15.37:0/3957866306 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed4011990 con 0x7f1edc0a0a80 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 10 monclient: get_monmap_and_config success 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 4 set_mon_vals no callback set 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 1 -- 172.21.15.37:0/3957866306 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4011d00 con 0x7f1edc0a0a80 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1ed9ffb700 10 monclient: dump: 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:48.457 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 10 monclient: shutdown 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 1 -- 172.21.15.37:0/3957866306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1edc0a0a80 msgr2=0x7f1edc13acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 1 --2- 172.21.15.37:0/3957866306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1edc0a0a80 0x7f1edc13acd0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4009850 tx=0x7f1ed4009b60 comp rx=0 tx=0).stop 2024-09-16T07:24:48.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 1 -- 172.21.15.37:0/3957866306 shutdown_connections 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 1 --2- 172.21.15.37:0/3957866306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1edc0a0a80 0x7f1edc13acd0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 1 -- 172.21.15.37:0/3957866306 >> 172.21.15.37:0/3957866306 conn(0x7f1edc0a3ea0 msgr2=0x7f1edc0a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 1 -- 172.21.15.37:0/3957866306 shutdown_connections 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 1 -- 172.21.15.37:0/3957866306 wait complete. 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 10 monclient: build_initial_monmap 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 10 monclient: monmap: 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:24:48.459 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:24:48.460 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:24:48.460 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:24:48.460 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:48.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 1 Processor -- start 2024-09-16T07:24:48.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 1 -- start start 2024-09-16T07:24:48.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.446+0000 7f1edbfff700 10 monclient: init 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1edc06b600 0x7f1edc12fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f1edc06b600 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1edc09a1c0 0x7f1edc130360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f1edc09a1c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1edc0a0a80 0x7f1edc138490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f1edc0a0a80 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc11a350 con 0x7f1edc0a0a80 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:48.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc119e90 con 0x7f1edc06b600 2024-09-16T07:24:48.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:48.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc139a00 con 0x7f1edc09a1c0 2024-09-16T07:24:48.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:48.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient(hunting): authenticate will time out at 1208.937157s 2024-09-16T07:24:48.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1edc09a1c0 0x7f1edc130360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:48.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edb7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1edc0a0a80 0x7f1edc138490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:48.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1edc09a1c0 0x7f1edc130360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40804/0 (socket says 172.21.15.37:40804) 2024-09-16T07:24:48.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 1 -- 172.21.15.37:0/559013229 learned_addr learned my addr 172.21.15.37:0/559013229 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:24:48.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 10 monclient(hunting): get_auth_request con 0x7f1edc09a1c0 auth_method 0 2024-09-16T07:24:48.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edb7fe700 10 monclient(hunting): get_auth_request con 0x7f1edc0a0a80 auth_method 0 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edaffd700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1edc06b600 0x7f1edc12fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edaffd700 10 monclient(hunting): get_auth_request con 0x7f1edc06b600 auth_method 0 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edaffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edb7fe700 1 -- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1edc0a0a80 msgr2=0x7f1edc138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T07:24:48.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edb7fe700 1 -- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1edc0a0a80 msgr2=0x7f1edc138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edb7fe700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1edc0a0a80 0x7f1edc138490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edb7fe700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1edc0a0a80 0x7f1edc138490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 10 monclient(hunting): handle_auth_done global_id 44489 payload 824 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 1 -- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1edc06b600 msgr2=0x7f1edc12fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1edc06b600 0x7f1edc12fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 1 -- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1edc0a0a80 msgr2=0x7f1edc138490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:24:48.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1edc0a0a80 0x7f1edc138490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 10 monclient: _finish_hunting 0 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 1 monclient: found mon.noname-c 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 1 -- 172.21.15.37:0/559013229 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed40094b0 con 0x7f1edc09a1c0 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edaffd700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1edc06b600 0x7f1edc12fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1eda7fc700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1edc09a1c0 0x7f1edc130360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00cbf0 tx=0x7f1ecc00cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 1 -- 172.21.15.37:0/559013229 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc00a790 con 0x7f1edc09a1c0 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 10 monclient: dump: 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:48.470 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 10 monclient: _finish_auth 0 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 10 monclient: _check_auth_tickets 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 1 -- 172.21.15.37:0/559013229 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ecc00a8f0 con 0x7f1edc09a1c0 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 5 monclient: authenticate success, global_id 44489 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:48.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient: _renew_subs 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 1 -- 172.21.15.37:0/559013229 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1edc11abf0 con 0x7f1edc09a1c0 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed8ff9700 4 set_mon_vals no callback set 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 1 -- 172.21.15.37:0/559013229 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc00ac60 con 0x7f1edc09a1c0 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient: _renew_subs 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1edbfff700 1 -- 172.21.15.37:0/559013229 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1edc0724c0 con 0x7f1edc09a1c0 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.450+0000 7f1ed37fe700 10 monclient: dump: 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:48.472 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.454+0000 7f1ed37fe700 1 -- 172.21.15.37:0/559013229 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f1ecc01b3b0 con 0x7f1edc09a1c0 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.454+0000 7f1edbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.454+0000 7f1edbfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.454+0000 7f1edbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.454+0000 7f1edbfff700 1 -- 172.21.15.37:0/559013229 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1edc11a490 con 0x7f1edc09a1c0 2024-09-16T07:24:48.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.454+0000 7f1ed37fe700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f1ebc035600 0x7f1ebc037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:48.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.454+0000 7f1ed37fe700 1 -- 172.21.15.37:0/559013229 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f1ecc047480 con 0x7f1edc09a1c0 2024-09-16T07:24:48.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.454+0000 7f1edaffd700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f1ebc035600 0x7f1ebc037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:48.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.454+0000 7f1edaffd700 10 monclient: get_auth_request con 0x7f1ebc035600 auth_method 0 2024-09-16T07:24:48.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.454+0000 7f1edaffd700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f1ebc035600 0x7f1ebc037ac0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1edc132ce0 tx=0x7f1ec4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:48.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.466+0000 7f1ed37fe700 1 -- 172.21.15.37:0/559013229 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1edc11a490 con 0x7f1edc09a1c0 2024-09-16T07:24:48.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.466+0000 7f1ed37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:48.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.466+0000 7f1ed37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:24:48.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.698+0000 7f1edbfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:24:48.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.698+0000 7f1edbfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:24:48.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.698+0000 7f1edbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:48.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.698+0000 7f1edbfff700 1 -- 172.21.15.37:0/559013229 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1edc11a490 con 0x7f1edc09a1c0 2024-09-16T07:24:48.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.698+0000 7f1ed37fe700 1 -- 172.21.15.37:0/559013229 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f1edc11a490 con 0x7f1edc09a1c0 2024-09-16T07:24:48.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.698+0000 7f1ed37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:24:48.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.698+0000 7f1ed37fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:24:48.703 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:24:48.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 -- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f1ebc035600 msgr2=0x7f1ebc037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:48.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f1ebc035600 0x7f1ebc037ac0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1edc132ce0 tx=0x7f1ec4009040 comp rx=0 tx=0).stop 2024-09-16T07:24:48.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 10 monclient: shutdown 2024-09-16T07:24:48.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:48.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 -- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1edc09a1c0 msgr2=0x7f1edc130360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:48.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1edc09a1c0 0x7f1edc130360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00cbf0 tx=0x7f1ecc00cc20 comp rx=0 tx=0).stop 2024-09-16T07:24:48.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 -- 172.21.15.37:0/559013229 shutdown_connections 2024-09-16T07:24:48.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1edc0a0a80 0x7f1edc138490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:48.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f1ebc035600 0x7f1ebc037ac0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:48.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1edc09a1c0 0x7f1edc130360 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:48.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 --2- 172.21.15.37:0/559013229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1edc06b600 0x7f1edc12fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:48.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 -- 172.21.15.37:0/559013229 >> 172.21.15.37:0/559013229 conn(0x7f1edc0a3ea0 msgr2=0x7f1edc09ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:48.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 -- 172.21.15.37:0/559013229 shutdown_connections 2024-09-16T07:24:48.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:48.702+0000 7f1edbfff700 1 -- 172.21.15.37:0/559013229 wait complete. 2024-09-16T07:24:51.718 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:24:51.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.834+0000 7f6e9ce9f700 10 monclient: get_monmap_and_config 2024-09-16T07:24:51.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.834+0000 7f6e9ce9f700 10 monclient: build_initial_monmap 2024-09-16T07:24:51.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.834+0000 7f6e9ce9f700 10 monclient: monmap: 2024-09-16T07:24:51.840 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:51.840 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:51.840 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:24:51.838554+0000 2024-09-16T07:24:51.840 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:24:51.838554+0000 2024-09-16T07:24:51.840 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:51.840 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:51.840 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:24:51.840 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:24:51.840 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:24:51.840 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:24:51.841 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:24:51.841 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:51.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.834+0000 7f6e9ce9f700 1 Processor -- start 2024-09-16T07:24:51.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.834+0000 7f6e9ce9f700 1 -- start start 2024-09-16T07:24:51.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.834+0000 7f6e9ce9f700 10 monclient: init 2024-09-16T07:24:51.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:51.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T07:24:51.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6e980a0a50 0x7f6e980a2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:51.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient(hunting): picked mon.noname-c con 0x7f6e980a0a50 addr v2:172.21.15.143:3300/0 2024-09-16T07:24:51.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e980a2c90 addr v1:172.21.15.37:6789/0 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a30c0 0x7f6e98139f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient(hunting): picked mon.noname-b con 0x7f6e980a30c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e980a0e30 con 0x7f6e980a2c90 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9809c570 con 0x7f6e980a0a50 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9809b890 con 0x7f6e980a30c0 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient(hunting): authenticate will time out at 1212.324435s 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e96ffd700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6e980a2c90 0x7f6e98139810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46632/0 (socket says 172.21.15.37:46632) 2024-09-16T07:24:51.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a30c0 0x7f6e98139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:51.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e96ffd700 1 -- 172.21.15.37:0/1427082992 learned_addr learned my addr 172.21.15.37:0/1427082992 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:24:51.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 10 monclient(hunting): get_auth_request con 0x7f6e980a30c0 auth_method 0 2024-09-16T07:24:51.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:51.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:51.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:51.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e977fe700 1 --2- 172.21.15.37:0/1427082992 >> v2:172.21.15.143:3300/0 conn(0x7f6e980a0a50 0x7f6e980a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:51.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:51.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:51.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:51.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e977fe700 10 monclient(hunting): get_auth_request con 0x7f6e980a0a50 auth_method 0 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 10 monclient(hunting): handle_auth_done global_id 44492 payload 274 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 1 -- 172.21.15.37:0/1427082992 >> v2:172.21.15.143:3300/0 conn(0x7f6e980a0a50 msgr2=0x7f6e980a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 1 --2- 172.21.15.37:0/1427082992 >> v2:172.21.15.143:3300/0 conn(0x7f6e980a0a50 0x7f6e980a2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 1 -- 172.21.15.37:0/1427082992 >> v1:172.21.15.37:6789/0 conn(0x7f6e980a2c90 legacy=0x7f6e98139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:51.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 10 monclient: _finish_hunting 0 2024-09-16T07:24:51.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 1 monclient: found mon.noname-b 2024-09-16T07:24:51.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:51.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:24:51.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 1 -- 172.21.15.37:0/1427082992 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98099be0 con 0x7f6e980a30c0 2024-09-16T07:24:51.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e97fff700 1 --2- 172.21.15.37:0/1427082992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a30c0 0x7f6e98139f20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009d90 tx=0x7f6e9000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebc975393f3bd178 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:51.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 1 -- 172.21.15.37:0/1427082992 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9000e070 con 0x7f6e980a30c0 2024-09-16T07:24:51.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 10 monclient: dump: 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:51.854 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 10 monclient: _finish_auth 0 2024-09-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 10 monclient: _check_auth_tickets 2024-09-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 5 monclient: authenticate success, global_id 44492 2024-09-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 1 -- 172.21.15.37:0/1427082992 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9000bb20 con 0x7f6e980a30c0 2024-09-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient: get_monmap_and_config success 2024-09-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 4 set_mon_vals no callback set 2024-09-16T07:24:51.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 1 -- 172.21.15.37:0/1427082992 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90012430 con 0x7f6e980a30c0 2024-09-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e967fc700 10 monclient: dump: 2024-09-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:51.856 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient: shutdown 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/1427082992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a30c0 msgr2=0x7f6e98139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 --2- 172.21.15.37:0/1427082992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a30c0 0x7f6e98139f20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009d90 tx=0x7f6e9000b330 comp rx=0 tx=0).stop 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/1427082992 shutdown_connections 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 --2- 172.21.15.37:0/1427082992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a30c0 0x7f6e98139f20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 --2- 172.21.15.37:0/1427082992 >> v2:172.21.15.143:3300/0 conn(0x7f6e980a0a50 0x7f6e980a2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:51.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/1427082992 >> 172.21.15.37:0/1427082992 conn(0x7f6e980a3ea0 msgr2=0x7f6e980a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/1427082992 shutdown_connections 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/1427082992 wait complete. 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient: build_initial_monmap 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 10 monclient: monmap: 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:24:51.858 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.838+0000 7f6e9ce9f700 1 Processor -- start 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 1 -- start start 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient: init 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e980a0a50 0x7f6e981ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient(hunting): picked mon.noname-a con 0x7f6e980a0a50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:24:51.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a2c90 0x7f6e98138af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient(hunting): picked mon.noname-c con 0x7f6e980a2c90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e980a30c0 0x7f6e98139030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient(hunting): picked mon.noname-b con 0x7f6e980a30c0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9809fef0 con 0x7f6e980a0a50 2024-09-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e981264b0 con 0x7f6e980a30c0 2024-09-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980724c0 con 0x7f6e980a2c90 2024-09-16T07:24:51.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient(hunting): authenticate will time out at 1212.328573s 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a2c90 0x7f6e98138af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e977fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e980a0a50 0x7f6e981ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a2c90 0x7f6e98138af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46380/0 (socket says 172.21.15.37:46380) 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 1 -- 172.21.15.37:0/410567556 learned_addr learned my addr 172.21.15.37:0/410567556 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 10 monclient(hunting): get_auth_request con 0x7f6e980a2c90 auth_method 0 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e977fe700 10 monclient(hunting): get_auth_request con 0x7f6e980a0a50 auth_method 0 2024-09-16T07:24:51.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e977fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e977fe700 1 -- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e980a0a50 msgr2=0x7f6e981ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e977fe700 1 -- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e980a0a50 msgr2=0x7f6e981ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e97fff700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e980a30c0 0x7f6e98139030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e977fe700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e980a0a50 0x7f6e981ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:24:51.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e97fff700 10 monclient(hunting): get_auth_request con 0x7f6e980a30c0 auth_method 0 2024-09-16T07:24:51.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:51.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e97fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:51.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e97fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:51.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e977fe700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e980a0a50 0x7f6e981ce080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:24:51.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e97fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:51.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e97fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:51.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e97fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:51.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 10 monclient(hunting): handle_auth_done global_id 44495 payload 824 2024-09-16T07:24:51.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 1 -- 172.21.15.37:0/410567556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e980a30c0 msgr2=0x7f6e98139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:51.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e980a30c0 0x7f6e98139030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 1 -- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e980a0a50 msgr2=0x7f6e981ce080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e980a0a50 0x7f6e981ce080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 10 monclient: _finish_hunting 0 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 1 monclient: found mon.noname-c 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 1 -- 172.21.15.37:0/410567556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e90009a70 con 0x7f6e980a2c90 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e96ffd700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a2c90 0x7f6e98138af0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800c730 tx=0x7f6e8800ca40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 1 -- 172.21.15.37:0/410567556 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88005d90 con 0x7f6e980a2c90 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:51.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 10 monclient: dump: 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:51.865 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 10 monclient: _finish_auth 0 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 10 monclient: _check_auth_tickets 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 5 monclient: authenticate success, global_id 44495 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient: _renew_subs 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/410567556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e98124aa0 con 0x7f6e980a2c90 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 1 -- 172.21.15.37:0/410567556 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e880070d0 con 0x7f6e980a2c90 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient: _renew_subs 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:51.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e95ffb700 4 set_mon_vals no callback set 2024-09-16T07:24:51.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/410567556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e98124170 con 0x7f6e980a2c90 2024-09-16T07:24:51.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 1 -- 172.21.15.37:0/410567556 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88007440 con 0x7f6e980a2c90 2024-09-16T07:24:51.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:51.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:51.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 10 monclient: dump: 2024-09-16T07:24:51.867 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:51.867 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:51.867 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:51.867 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:51.867 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 1 -- 172.21.15.37:0/410567556 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f6e880075a0 con 0x7f6e980a2c90 2024-09-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6e84035600 0x7f6e84037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e7ffff700 1 -- 172.21.15.37:0/410567556 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f6e88046ac0 con 0x7f6e980a2c90 2024-09-16T07:24:51.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:51.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:51.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/410567556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e98125f50 con 0x7f6e980a2c90 2024-09-16T07:24:51.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.842+0000 7f6e977fe700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6e84035600 0x7f6e84037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:51.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.846+0000 7f6e977fe700 10 monclient: get_auth_request con 0x7f6e84035600 auth_method 0 2024-09-16T07:24:51.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.846+0000 7f6e977fe700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6e84035600 0x7f6e84037ac0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c009430 tx=0x7f6e8c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:51.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.850+0000 7f6e7ffff700 1 -- 172.21.15.37:0/410567556 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6e98125f50 con 0x7f6e980a2c90 2024-09-16T07:24:51.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.850+0000 7f6e7ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:51.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:51.850+0000 7f6e7ffff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:24:52.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.090+0000 7f6e9ce9f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:24:52.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.090+0000 7f6e9ce9f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:24:52.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.090+0000 7f6e9ce9f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:52.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.090+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/410567556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6e98125f50 con 0x7f6e980a2c90 2024-09-16T07:24:52.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.090+0000 7f6e7ffff700 1 -- 172.21.15.37:0/410567556 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1538 (secure 0 0 0) 0x7f6e98125f50 con 0x7f6e980a2c90 2024-09-16T07:24:52.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.090+0000 7f6e7ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:24:52.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.090+0000 7f6e7ffff700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:24:52.095 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:24:52.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/410567556 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6e84035600 msgr2=0x7f6e84037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:52.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6e84035600 0x7f6e84037ac0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c009430 tx=0x7f6e8c008040 comp rx=0 tx=0).stop 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 10 monclient: shutdown 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a2c90 msgr2=0x7f6e98138af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a2c90 0x7f6e98138af0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800c730 tx=0x7f6e8800ca40 comp rx=0 tx=0).stop 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/410567556 shutdown_connections 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e980a30c0 0x7f6e98139030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6e84035600 0x7f6e84037ac0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e980a2c90 0x7f6e98138af0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 --2- 172.21.15.37:0/410567556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e980a0a50 0x7f6e981ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/410567556 >> 172.21.15.37:0/410567556 conn(0x7f6e980a3ea0 msgr2=0x7f6e981c1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/410567556 shutdown_connections 2024-09-16T07:24:52.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:52.094+0000 7f6e9ce9f700 1 -- 172.21.15.37:0/410567556 wait complete. 2024-09-16T07:24:55.110 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:24:55.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient: get_monmap_and_config 2024-09-16T07:24:55.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient: build_initial_monmap 2024-09-16T07:24:55.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient: monmap: 2024-09-16T07:24:55.232 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:55.232 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:55.232 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:24:55.231514+0000 2024-09-16T07:24:55.232 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:24:55.231514+0000 2024-09-16T07:24:55.232 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:55.232 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:55.232 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:24:55.233 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:24:55.233 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:24:55.233 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:24:55.233 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:24:55.233 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:55.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 Processor -- start 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 -- start start 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient: init 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3c0c0a2640 0x7f3c0c0a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient(hunting): picked mon.noname-a con 0x7f3c0c0a2640 addr v2:172.21.15.37:3300/0 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 0x7f3c0c135400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient(hunting): picked mon.noname-b con 0x7f3c0c0a2f60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c0c135940 addr v1:172.21.15.37:6789/0 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c0a3770 con 0x7f3c0c135940 2024-09-16T07:24:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0c06cbe0 con 0x7f3c0c0a2640 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c09fb80 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient(hunting): authenticate will time out at 1215.717353s 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3c0c0a2640 0x7f3c0c0a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 0x7f3c0c135400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3c0c0a2640 0x7f3c0c0a2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37992/0 (socket says 172.21.15.37:37992) 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 1 -- 172.21.15.37:0/2357227029 learned_addr learned my addr 172.21.15.37:0/2357227029 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 10 monclient(hunting): get_auth_request con 0x7f3c0c0a2640 auth_method 0 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c121a6700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3c0c135940 0x7f3c0c13c650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46640/0 (socket says 172.21.15.37:46640) 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 10 monclient(hunting): get_auth_request con 0x7f3c0c0a2f60 auth_method 0 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 1 -- 172.21.15.37:0/2357227029 >> v2:172.21.15.37:3300/0 conn(0x7f3c0c0a2640 msgr2=0x7f3c0c0a2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 1 -- 172.21.15.37:0/2357227029 >> v2:172.21.15.37:3300/0 conn(0x7f3c0c0a2640 msgr2=0x7f3c0c0a2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 1 --2- 172.21.15.37:0/2357227029 >> v2:172.21.15.37:3300/0 conn(0x7f3c0c0a2640 0x7f3c0c0a2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c119a5700 1 --2- 172.21.15.37:0/2357227029 >> v2:172.21.15.37:3300/0 conn(0x7f3c0c0a2640 0x7f3c0c0a2a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 10 monclient(hunting): handle_auth_done global_id 44498 payload 274 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 1 -- 172.21.15.37:0/2357227029 >> v2:172.21.15.37:3300/0 conn(0x7f3c0c0a2640 msgr2=0x7f3c0c0a2a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 1 --2- 172.21.15.37:0/2357227029 >> v2:172.21.15.37:3300/0 conn(0x7f3c0c0a2640 0x7f3c0c0a2a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 1 -- 172.21.15.37:0/2357227029 >> v1:172.21.15.37:6789/0 conn(0x7f3c0c135940 legacy=0x7f3c0c13c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 10 monclient: _finish_hunting 0 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 1 monclient: found mon.noname-b 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 1 -- 172.21.15.37:0/2357227029 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c0a1f80 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c111a4700 1 --2- 172.21.15.37:0/2357227029 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 0x7f3c0c135400 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009850 tx=0x7f3c08009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95f33a39cf1ce81f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 1 -- 172.21.15.37:0/2357227029 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08011830 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:24:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 10 monclient: dump: 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 10 monclient: _finish_auth 0 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 10 monclient: _check_auth_tickets 2024-09-16T07:24:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 5 monclient: authenticate success, global_id 44498 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 1 -- 172.21.15.37:0/2357227029 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c08011990 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient: get_monmap_and_config success 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 4 set_mon_vals no callback set 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 1 -- 172.21.15.37:0/2357227029 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08011d20 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c109a3700 10 monclient: dump: 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 10 monclient: shutdown 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 -- 172.21.15.37:0/2357227029 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 msgr2=0x7f3c0c135400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 --2- 172.21.15.37:0/2357227029 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 0x7f3c0c135400 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009850 tx=0x7f3c08009b60 comp rx=0 tx=0).stop 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 -- 172.21.15.37:0/2357227029 shutdown_connections 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 --2- 172.21.15.37:0/2357227029 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 0x7f3c0c135400 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 --2- 172.21.15.37:0/2357227029 >> v2:172.21.15.37:3300/0 conn(0x7f3c0c0a2640 0x7f3c0c0a2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 -- 172.21.15.37:0/2357227029 >> 172.21.15.37:0/2357227029 conn(0x7f3c0c0a8270 msgr2=0x7f3c0c0a4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 -- 172.21.15.37:0/2357227029 shutdown_connections 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.230+0000 7f3c129a7700 1 -- 172.21.15.37:0/2357227029 wait complete. 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient: build_initial_monmap 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient: monmap: 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:24:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:24:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:24:55.246 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 1 Processor -- start 2024-09-16T07:24:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 1 -- start start 2024-09-16T07:24:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient: init 2024-09-16T07:24:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:24:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c0c0a2640 0x7f3c0c1ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient(hunting): picked mon.noname-a con 0x7f3c0c0a2640 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:24:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 0x7f3c0c134880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient(hunting): picked mon.noname-c con 0x7f3c0c0a2f60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c0c135940 0x7f3c0c134dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient(hunting): picked mon.noname-b con 0x7f3c0c135940 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c125a00 con 0x7f3c0c0a2640 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c124450 con 0x7f3c0c135940 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c0a6d70 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c129a7700 10 monclient(hunting): authenticate will time out at 1215.723311s 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c0c0a2640 0x7f3c0c1ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c0c0a2640 0x7f3c0c1ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37996/0 (socket says 172.21.15.37:37996) 2024-09-16T07:24:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 1 -- 172.21.15.37:0/3007134456 learned_addr learned my addr 172.21.15.37:0/3007134456 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 10 monclient(hunting): get_auth_request con 0x7f3c0c0a2640 auth_method 0 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c111a4700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 0x7f3c0c134880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 1 -- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c0c0a2640 msgr2=0x7f3c0c1ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 1 -- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c0c0a2640 msgr2=0x7f3c0c1ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c0c0a2640 0x7f3c0c1ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c119a5700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c0c0a2640 0x7f3c0c1ca250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c111a4700 10 monclient(hunting): get_auth_request con 0x7f3c0c0a2f60 auth_method 0 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c111a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c111a4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c111a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c121a6700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c0c135940 0x7f3c0c134dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c111a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c111a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c111a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c121a6700 10 monclient(hunting): get_auth_request con 0x7f3c0c135940 auth_method 0 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c121a6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c121a6700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.234+0000 7f3c121a6700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c121a6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c121a6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c121a6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 10 monclient(hunting): handle_auth_done global_id 44501 payload 824 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 1 -- 172.21.15.37:0/3007134456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c0c135940 msgr2=0x7f3c0c134dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c0c135940 0x7f3c0c134dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 1 -- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c0c0a2640 msgr2=0x7f3c0c1ca250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c0c0a2640 0x7f3c0c1ca250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 10 monclient: _finish_hunting 0 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 1 monclient: found mon.noname-c 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 1 -- 172.21.15.37:0/3007134456 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c080094b0 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c111a4700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 0x7f3c0c134880 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c08016000 tx=0x7f3c08004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 1 -- 172.21.15.37:0/3007134456 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c080036c0 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 10 monclient: dump: 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 10 monclient: _finish_auth 0 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 1 -- 172.21.15.37:0/3007134456 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c08003820 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 1 -- 172.21.15.37:0/3007134456 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08003bd0 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 5 monclient: authenticate success, global_id 44501 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 10 monclient: _renew_subs 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 1 -- 172.21.15.37:0/3007134456 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0c06d750 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfffff700 4 set_mon_vals no callback set 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 10 monclient: dump: 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 10 monclient: _renew_subs 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 1 -- 172.21.15.37:0/3007134456 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0c123f40 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 1 -- 172.21.15.37:0/3007134456 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f3c08021070 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c129a7700 1 -- 172.21.15.37:0/3007134456 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0c0a0080 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3bf0035630 0x7f3bf0037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3bfe7fc700 1 -- 172.21.15.37:0/3007134456 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f3c08048f40 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c119a5700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3bf0035630 0x7f3bf0037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.238+0000 7f3c119a5700 10 monclient: get_auth_request con 0x7f3bf0035630 auth_method 0 2024-09-16T07:24:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.242+0000 7f3c119a5700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3bf0035630 0x7f3bf0037af0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3c00006b40 tx=0x7f3c00008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.250+0000 7f3bfe7fc700 1 -- 172.21.15.37:0/3007134456 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3c080247f0 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.250+0000 7f3bfe7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.250+0000 7f3bfe7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:24:55.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.482+0000 7f3c129a7700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:24:55.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.482+0000 7f3c129a7700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:24:55.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.482+0000 7f3c129a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:55.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.482+0000 7f3c129a7700 1 -- 172.21.15.37:0/3007134456 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3c0c0a0580 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.482+0000 7f3bfe7fc700 1 -- 172.21.15.37:0/3007134456 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f3c08015070 con 0x7f3c0c0a2f60 2024-09-16T07:24:55.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.482+0000 7f3bfe7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:24:55.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.482+0000 7f3bfe7fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:24:55.487 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":101,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:24:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 -- 172.21.15.37:0/3007134456 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3bf0035630 msgr2=0x7f3bf0037af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:55.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3bf0035630 0x7f3bf0037af0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3c00006b40 tx=0x7f3c00008040 comp rx=0 tx=0).stop 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 10 monclient: shutdown 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 -- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 msgr2=0x7f3c0c134880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 0x7f3c0c134880 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3c08016000 tx=0x7f3c08004770 comp rx=0 tx=0).stop 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 -- 172.21.15.37:0/3007134456 shutdown_connections 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c0c135940 0x7f3c0c134dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3bf0035630 0x7f3bf0037af0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0c0a2f60 0x7f3c0c134880 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 --2- 172.21.15.37:0/3007134456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c0c0a2640 0x7f3c0c1ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 -- 172.21.15.37:0/3007134456 >> 172.21.15.37:0/3007134456 conn(0x7f3c0c0a8270 msgr2=0x7f3c0c09bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 -- 172.21.15.37:0/3007134456 shutdown_connections 2024-09-16T07:24:55.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:55.486+0000 7f3c129a7700 1 -- 172.21.15.37:0/3007134456 wait complete. 2024-09-16T07:24:58.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:24:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient: get_monmap_and_config 2024-09-16T07:24:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient: build_initial_monmap 2024-09-16T07:24:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient: monmap: 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:24:58.623669+0000 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:24:58.623669+0000 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:24:58.625 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 1 Processor -- start 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 1 -- start start 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient: init 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6cc40254f0 0x7f6cc40c0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient(hunting): picked mon.noname-b con 0x7f6cc40254f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6cc4022f10 addr v1:172.21.15.143:6789/0 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cc4023310 addr v1:172.21.15.37:6789/0 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc4020380 con 0x7f6cc4023310 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc40258d0 con 0x7f6cc4022f10 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc401ebe0 con 0x7f6cc40254f0 2024-09-16T07:24:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient(hunting): authenticate will time out at 1219.109714s 2024-09-16T07:24:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6cc40254f0 0x7f6cc40c0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd24cc700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6cc4023310 0x7f6cc401dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46654/0 (socket says 172.21.15.37:46654) 2024-09-16T07:24:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd24cc700 1 -- 172.21.15.37:0/3090052573 learned_addr learned my addr 172.21.15.37:0/3090052573 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:24:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 10 monclient(hunting): get_auth_request con 0x7f6cc40254f0 auth_method 0 2024-09-16T07:24:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 1 -- 172.21.15.37:0/3090052573 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310544919 0 0) 0x7f6cc40258d0 con 0x7f6cc4022f10 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient(hunting): _negotiate 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient(hunting): my global_id is 6268 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 1 -- 172.21.15.37:0/3090052573 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb80035d0 con 0x7f6cc4022f10 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 10 monclient(hunting): handle_auth_done global_id 44504 payload 274 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 1 -- 172.21.15.37:0/3090052573 >> v1:172.21.15.143:6789/0 conn(0x7f6cc4022f10 legacy=0x7f6cc401d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 1 -- 172.21.15.37:0/3090052573 >> v1:172.21.15.37:6789/0 conn(0x7f6cc4023310 legacy=0x7f6cc401dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 10 monclient: _finish_hunting 0 2024-09-16T07:24:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 1 monclient: found mon.noname-b 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 1 -- 172.21.15.37:0/3090052573 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc401e4a0 con 0x7f6cc40254f0 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd1ccb700 1 --2- 172.21.15.37:0/3090052573 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6cc40254f0 0x7f6cc40c0100 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6cc401ed20 tx=0x7f6cbc009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f91180b5f403e73 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 1 -- 172.21.15.37:0/3090052573 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc004230 con 0x7f6cc40254f0 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient: dump: 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:58.634 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient: _finish_auth 0 2024-09-16T07:24:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient: _check_auth_tickets 2024-09-16T07:24:58.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 1 -- 172.21.15.37:0/3090052573 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc004390 con 0x7f6cc40254f0 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 5 monclient: authenticate success, global_id 44504 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 1 -- 172.21.15.37:0/3090052573 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc0036c0 con 0x7f6cc40254f0 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 10 monclient: get_monmap_and_config success 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd2ccd700 4 set_mon_vals no callback set 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.622+0000 7f6cd0cc9700 10 monclient: dump: 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:58.635 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 10 monclient: shutdown 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/3090052573 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6cc40254f0 msgr2=0x7f6cc40c0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 --2- 172.21.15.37:0/3090052573 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6cc40254f0 0x7f6cc40c0100 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6cc401ed20 tx=0x7f6cbc009b60 comp rx=0 tx=0).stop 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/3090052573 shutdown_connections 2024-09-16T07:24:58.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 --2- 172.21.15.37:0/3090052573 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6cc40254f0 0x7f6cc40c0100 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/3090052573 >> 172.21.15.37:0/3090052573 conn(0x7f6cc40064b0 msgr2=0x7f6cc40068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/3090052573 shutdown_connections 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/3090052573 wait complete. 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 10 monclient: build_initial_monmap 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 10 monclient: monmap: 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:24:58.637 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:24:58.638 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:24:58.638 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:58.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 Processor -- start 2024-09-16T07:24:58.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 -- start start 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 10 monclient: init 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6cc4022f10 0x7f6cc4154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 10 monclient(hunting): picked mon.noname-a con 0x7f6cc4022f10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4023310 0x7f6cc4155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.626+0000 7f6cd2ccd700 10 monclient(hunting): picked mon.noname-b con 0x7f6cc4023310 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6cc40254f0 0x7f6cc40aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 monclient(hunting): picked mon.noname-c con 0x7f6cc40254f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:58.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc4020870 con 0x7f6cc4022f10 2024-09-16T07:24:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc4028f00 con 0x7f6cc4023310 2024-09-16T07:24:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc4029450 con 0x7f6cc40254f0 2024-09-16T07:24:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 monclient(hunting): authenticate will time out at 1219.116070s 2024-09-16T07:24:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6cc4022f10 0x7f6cc4154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6cc4022f10 0x7f6cc4154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:38012/0 (socket says 172.21.15.37:38012) 2024-09-16T07:24:58.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 1 -- 172.21.15.37:0/961864354 learned_addr learned my addr 172.21.15.37:0/961864354 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 10 monclient(hunting): get_auth_request con 0x7f6cc4022f10 auth_method 0 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd14ca700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4023310 0x7f6cc4155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd14ca700 10 monclient(hunting): get_auth_request con 0x7f6cc4023310 auth_method 0 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd14ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd14ca700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd14ca700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:58.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd14ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd14ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd14ca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 10 monclient(hunting): handle_auth_done global_id 54099 payload 824 2024-09-16T07:24:58.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 1 -- 172.21.15.37:0/961864354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6cc40254f0 msgr2=0x7f6cc40aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6cc40254f0 0x7f6cc40aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 1 -- 172.21.15.37:0/961864354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4023310 msgr2=0x7f6cc4155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4023310 0x7f6cc4155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 10 monclient: _finish_hunting 0 2024-09-16T07:24:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 1 monclient: found mon.noname-a 2024-09-16T07:24:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:24:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 1 -- 172.21.15.37:0/961864354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc0094b0 con 0x7f6cc4022f10 2024-09-16T07:24:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd1ccb700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6cc4022f10 0x7f6cc4154e10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc000c00 tx=0x7f6cbc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:58.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 1 -- 172.21.15.37:0/961864354 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc011630 con 0x7f6cc4022f10 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 10 monclient: dump: 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:58.648 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 10 monclient: _finish_auth 0 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 10 monclient: _check_auth_tickets 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 1 -- 172.21.15.37:0/961864354 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc0117b0 con 0x7f6cc4022f10 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 1 -- 172.21.15.37:0/961864354 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc011b60 con 0x7f6cc4022f10 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc27fc700 10 monclient: dump: 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:58.649 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 5 monclient: authenticate success, global_id 54099 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 monclient: _renew_subs 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:58.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/961864354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc40b02e0 con 0x7f6cc4022f10 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cc3fff700 4 set_mon_vals no callback set 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 monclient: _renew_subs 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/961864354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc40ae720 con 0x7f6cc4022f10 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.630+0000 7f6cd2ccd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.634+0000 7f6cd2ccd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.634+0000 7f6cd2ccd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.634+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/961864354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc401ef20 con 0x7f6cc4022f10 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.634+0000 7f6cc27fc700 1 -- 172.21.15.37:0/961864354 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f6cbc021070 con 0x7f6cc4022f10 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.634+0000 7f6cc27fc700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6cb0035600 0x7f6cb0037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.634+0000 7f6cc27fc700 1 -- 172.21.15.37:0/961864354 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f6cbc049030 con 0x7f6cc4022f10 2024-09-16T07:24:58.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.634+0000 7f6cd14ca700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6cb0035600 0x7f6cb0037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.634+0000 7f6cd14ca700 10 monclient: get_auth_request con 0x7f6cb0035600 auth_method 0 2024-09-16T07:24:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.634+0000 7f6cd14ca700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6cb0035600 0x7f6cb0037ac0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40a3db0 tx=0x7f6cc8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.642+0000 7f6cc27fc700 1 -- 172.21.15.37:0/961864354 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6cc401ef20 con 0x7f6cc4022f10 2024-09-16T07:24:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.642+0000 7f6cc27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:58.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.642+0000 7f6cc27fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:24:58.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.878+0000 7f6cd2ccd700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:24:58.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.878+0000 7f6cd2ccd700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:24:58.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.878+0000 7f6cd2ccd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:58.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.878+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/961864354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6cc401ef20 con 0x7f6cc4022f10 2024-09-16T07:24:58.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cc27fc700 1 -- 172.21.15.37:0/961864354 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f6cc401ef20 con 0x7f6cc4022f10 2024-09-16T07:24:58.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cc27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:24:58.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cc27fc700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:24:58.885 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:24:58.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/961864354 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6cb0035600 msgr2=0x7f6cb0037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:58.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6cb0035600 0x7f6cb0037ac0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40a3db0 tx=0x7f6cc8009040 comp rx=0 tx=0).stop 2024-09-16T07:24:58.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 10 monclient: shutdown 2024-09-16T07:24:58.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:58.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/961864354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6cc4022f10 msgr2=0x7f6cc4154e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:58.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6cc4022f10 0x7f6cc4154e10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc000c00 tx=0x7f6cbc004770 comp rx=0 tx=0).stop 2024-09-16T07:24:58.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/961864354 shutdown_connections 2024-09-16T07:24:58.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6cb0035600 0x7f6cb0037ac0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:58.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6cc40254f0 0x7f6cc40aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:58.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc4023310 0x7f6cc4155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:58.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 --2- 172.21.15.37:0/961864354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6cc4022f10 0x7f6cc4154e10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:58.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/961864354 >> 172.21.15.37:0/961864354 conn(0x7f6cc40064b0 msgr2=0x7f6cc4023fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:58.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/961864354 shutdown_connections 2024-09-16T07:24:58.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:58.882+0000 7f6cd2ccd700 1 -- 172.21.15.37:0/961864354 wait complete. 2024-09-16T07:24:58.899 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T07:24:58.900 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T07:24:58.900 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T07:24:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.006+0000 7ff06a658700 10 monclient: get_monmap_and_config 2024-09-16T07:24:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.006+0000 7ff06a658700 10 monclient: build_initial_monmap 2024-09-16T07:24:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.006+0000 7ff06a658700 10 monclient: monmap: 2024-09-16T07:24:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:24:59.010317+0000 2024-09-16T07:24:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:24:59.010317+0000 2024-09-16T07:24:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:24:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:24:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:24:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:24:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:24:59.012 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.006+0000 7ff06a658700 1 Processor -- start 2024-09-16T07:24:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.006+0000 7ff06a658700 1 -- start start 2024-09-16T07:24:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.006+0000 7ff06a658700 10 monclient: init 2024-09-16T07:24:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff0640a2750 0x7ff0640a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient(hunting): picked mon.noname-a con 0x7ff0640a2750 addr v2:172.21.15.37:3300/0 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0640a3070 addr v1:172.21.15.37:6789/0 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff064133730 0x7ff06413b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient(hunting): picked mon.noname-c con 0x7ff064133730 addr v2:172.21.15.143:3300/0 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff064099be0 con 0x7ff0640a3070 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06409de40 con 0x7ff0640a2750 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06409d530 con 0x7ff064133730 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient(hunting): authenticate will time out at 1219.496195s 2024-09-16T07:24:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff068e55700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff0640a3070 0x7ff064133020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46664/0 (socket says 172.21.15.37:46664) 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff068e55700 1 -- 172.21.15.37:0/1977670183 learned_addr learned my addr 172.21.15.37:0/1977670183 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 1 --2- 172.21.15.37:0/1977670183 >> v2:172.21.15.37:3300/0 conn(0x7ff0640a2750 0x7ff0640a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 10 monclient(hunting): get_auth_request con 0x7ff0640a2750 auth_method 0 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069e57700 1 --2- 172.21.15.37:0/1977670183 >> v2:172.21.15.143:3300/0 conn(0x7ff064133730 0x7ff06413b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069e57700 10 monclient(hunting): get_auth_request con 0x7ff064133730 auth_method 0 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069e57700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069e57700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069e57700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 1 -- 172.21.15.37:0/1977670183 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248116221 0 0) 0x7ff064099be0 con 0x7ff0640a3070 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient(hunting): _negotiate 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient(hunting): my global_id is 54102 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 1 -- 172.21.15.37:0/1977670183 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0500035d0 con 0x7ff0640a3070 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069e57700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069e57700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069e57700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 10 monclient(hunting): handle_auth_done global_id 54105 payload 274 2024-09-16T07:24:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 1 -- 172.21.15.37:0/1977670183 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2936532446 0 0) 0x7ff0500035d0 con 0x7ff0640a3070 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 1 -- 172.21.15.37:0/1977670183 >> v2:172.21.15.143:3300/0 conn(0x7ff064133730 msgr2=0x7ff06413b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 1 --2- 172.21.15.37:0/1977670183 >> v2:172.21.15.143:3300/0 conn(0x7ff064133730 0x7ff06413b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 1 -- 172.21.15.37:0/1977670183 >> v1:172.21.15.37:6789/0 conn(0x7ff0640a3070 legacy=0x7ff064133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 10 monclient: _finish_hunting 0 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 1 monclient: found mon.noname-a 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 1 -- 172.21.15.37:0/1977670183 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06409a020 con 0x7ff0640a2750 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff069656700 1 --2- 172.21.15.37:0/1977670183 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a2750 0x7ff0640a2b30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff0640711d0 tx=0x7ff054009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70987aa692d8635e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 1 -- 172.21.15.37:0/1977670183 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060001670 con 0x7ff0640a2750 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:24:59.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient: dump: 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient: _finish_auth 0 2024-09-16T07:24:59.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient: _check_auth_tickets 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 5 monclient: authenticate success, global_id 54105 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 1 -- 172.21.15.37:0/1977670183 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff054004030 con 0x7ff0640a2750 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient: get_monmap_and_config success 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 4 set_mon_vals no callback set 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 1 -- 172.21.15.37:0/1977670183 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0540043a0 con 0x7ff0640a2750 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff05bfff700 10 monclient: dump: 2024-09-16T07:24:59.032 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient: shutdown 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:59.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 -- 172.21.15.37:0/1977670183 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a2750 msgr2=0x7ff0640a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 --2- 172.21.15.37:0/1977670183 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a2750 0x7ff0640a2b30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff0640711d0 tx=0x7ff054009b30 comp rx=0 tx=0).stop 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 -- 172.21.15.37:0/1977670183 shutdown_connections 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 --2- 172.21.15.37:0/1977670183 >> v2:172.21.15.143:3300/0 conn(0x7ff064133730 0x7ff06413b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 --2- 172.21.15.37:0/1977670183 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a2750 0x7ff0640a2b30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 -- 172.21.15.37:0/1977670183 >> 172.21.15.37:0/1977670183 conn(0x7ff0640a3ea0 msgr2=0x7ff0640a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 -- 172.21.15.37:0/1977670183 shutdown_connections 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 1 -- 172.21.15.37:0/1977670183 wait complete. 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient: build_initial_monmap 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.010+0000 7ff06a658700 10 monclient: monmap: 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:24:59.034 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 1 Processor -- start 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 1 -- start start 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient: init 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0640a2750 0x7ff064132360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient(hunting): picked mon.noname-c con 0x7ff0640a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 0x7ff0641328a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient(hunting): picked mon.noname-a con 0x7ff0640a3070 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff064133730 0x7ff06411be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient(hunting): picked mon.noname-b con 0x7ff064133730 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff064133120 con 0x7ff0640a3070 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06406d080 con 0x7ff064133730 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06411fa40 con 0x7ff0640a2750 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient(hunting): authenticate will time out at 1219.502395s 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0640a2750 0x7ff064132360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff068e55700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 0x7ff0641328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0640a2750 0x7ff064132360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46436/0 (socket says 172.21.15.37:46436) 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff068e55700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 0x7ff0641328a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:38032/0 (socket says 172.21.15.37:38032) 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 1 -- 172.21.15.37:0/1971004710 learned_addr learned my addr 172.21.15.37:0/1971004710 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 10 monclient(hunting): get_auth_request con 0x7ff0640a2750 auth_method 0 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff068e55700 10 monclient(hunting): get_auth_request con 0x7ff0640a3070 auth_method 0 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff068e55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff068e55700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff068e55700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff068e55700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 msgr2=0x7ff0641328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff068e55700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 msgr2=0x7ff0641328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff068e55700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 0x7ff0641328a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff068e55700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 0x7ff0641328a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069e57700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff064133730 0x7ff06411be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069e57700 10 monclient(hunting): get_auth_request con 0x7ff064133730 auth_method 0 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069e57700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069e57700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069e57700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069e57700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069e57700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069e57700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 10 monclient(hunting): handle_auth_done global_id 44507 payload 824 2024-09-16T07:24:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff064133730 msgr2=0x7ff06411be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff064133730 0x7ff06411be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 msgr2=0x7ff0641328a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:24:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 0x7ff0641328a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 10 monclient: _finish_hunting 0 2024-09-16T07:24:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 1 monclient: found mon.noname-c 2024-09-16T07:24:59.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0540094b0 con 0x7ff0640a2750 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff069656700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0640a2750 0x7ff064132360 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff0540090b0 tx=0x7ff054004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054003af0 con 0x7ff0640a2750 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff059ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff059ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff059ffb700 10 monclient: dump: 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff059ffb700 10 monclient: _finish_auth 0 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff059ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff059ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05401f070 con 0x7ff0640a2750 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 5 monclient: authenticate success, global_id 44507 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient: _renew_subs 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.014+0000 7ff06a658700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06411e590 con 0x7ff0640a2750 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054003e90 con 0x7ff0640a2750 2024-09-16T07:24:59.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: dump: 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff05b7fe700 4 set_mon_vals no callback set 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff06a658700 10 monclient: _renew_subs 2024-09-16T07:24:59.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0640a2750 msgr2=0x7ff064132360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff06a658700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0640a2750 msgr2=0x7ff064132360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff06a658700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06411e310 con 0x7ff0640a2750 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0640a2750 0x7ff064132360 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff0540090b0 tx=0x7ff054004770 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0640a2750 0x7ff064132360 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff0540090b0 tx=0x7ff054004770 comp rx=0 tx=0).stop 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0640a2750 msgr2=0x7ff064132360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0640a2750 0x7ff064132360 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff064133730 0x7ff048005920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient(hunting): picked mon.b con 0x7ff064133730 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:24:59.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0480062b0 0x7ff048008740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient(hunting): picked mon.c con 0x7ff0480062b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 0x7ff04800b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient(hunting): picked mon.a con 0x7ff0640a3070 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04800bc70 con 0x7ff0640a3070 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04800bdb0 con 0x7ff064133730 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04800bf20 con 0x7ff0480062b0 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 0x7ff04800b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient(hunting): get_auth_request con 0x7ff0640a3070 auth_method 0 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff064133730 0x7ff048005920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): get_auth_request con 0x7ff064133730 auth_method 0 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:24:59.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient(hunting): handle_auth_done global_id 44507 payload 293 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0480062b0 msgr2=0x7ff048008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff06a658700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0480062b0 0x7ff048008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff064133730 msgr2=0x7ff048005920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff064133730 0x7ff048005920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient: _finish_hunting 0 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 1 monclient: found mon.a 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7ff04800c210 con 0x7ff0640a3070 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient: _finish_auth 0 2024-09-16T07:24:59.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 10 monclient: _check_auth_tickets 2024-09-16T07:24:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff06a658700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641c8180 0x7ff064132360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff06a658700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff06a658700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff064067530 con 0x7ff0641c8180 2024-09-16T07:24:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641c8180 0x7ff064132360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient: get_auth_request con 0x7ff0641c8180 auth_method 0 2024-09-16T07:24:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069e57700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 0x7ff04800b260 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009f80 tx=0x7ff054008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c00e040 con 0x7ff0640a3070 2024-09-16T07:24:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:24:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: dump: 2024-09-16T07:24:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.048 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c00f040 con 0x7ff0640a3070 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 10 monclient(hunting): handle_auth_done global_id 54108 payload 824 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff05affd700 4 set_mon_vals no callback set 2024-09-16T07:24:59.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0640999e0 con 0x7ff0641c8180 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff069656700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641c8180 0x7ff064132360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff054009b70 tx=0x7ff054009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7ff05c011d40 con 0x7ff0640a3070 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff04803e200 0x7ff0480406c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.0 v2:172.21.15.37:3300/0 4 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7ff05c045e60 con 0x7ff0640a3070 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054034070 con 0x7ff0641c8180 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: dump: 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.050 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7ff05401a7b0 con 0x7ff0641c8180 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641c8180 msgr2=0x7ff064132360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641c8180 0x7ff064132360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff054009b70 tx=0x7ff054009240 comp rx=0 tx=0).stop 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641c8180 msgr2=0x7ff064132360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff059ffb700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641c8180 0x7ff064132360 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff054009b70 tx=0x7ff054009240 comp rx=0 tx=0).stop 2024-09-16T07:24:59.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff068e55700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff04803e200 0x7ff0480406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.018+0000 7ff068e55700 10 monclient: get_auth_request con 0x7ff04803e200 auth_method 0 2024-09-16T07:24:59.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.022+0000 7ff068e55700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff04803e200 0x7ff0480406c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff06411d7f0 tx=0x7ff060001c80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:24:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641328a0 0x7ff06411be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff064072550 con 0x7ff0641328a0 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641328a0 0x7ff06411be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 10 monclient: get_auth_request con 0x7ff0641328a0 auth_method 0 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 10 monclient(hunting): handle_auth_done global_id 54111 payload 824 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 1 -- 172.21.15.37:0/1971004710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff0640a3480 con 0x7ff0641328a0 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069656700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641328a0 0x7ff06411be00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff054006cb0 tx=0x7ff05401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05402fc40 con 0x7ff0641328a0 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 10 monclient: dump: 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4453 (secure 0 0 0) 0x7ff054040300 con 0x7ff0641328a0 2024-09-16T07:24:59.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:24:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:24:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641328a0 msgr2=0x7ff06411be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641328a0 0x7ff06411be00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff054006cb0 tx=0x7ff05401f000 comp rx=0 tx=0).stop 2024-09-16T07:24:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641328a0 msgr2=0x7ff06411be00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:24:59.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff059ffb700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0641328a0 0x7ff06411be00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.056 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:24:59.056 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:24:59.056 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:24:59.056 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:24:59.056 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 28, 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:24:59.057 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:24:59.058 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:24:59.059 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:24:59.060 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.061 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:24:59.062 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:24:59.063 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.064 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:24:59.065 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:24:59.066 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.067 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:24:59.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.068 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:24:59.068 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:24:59.068 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:24:59.068 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.068 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.068 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:24:59.068 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:24:59.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff04803e200 msgr2=0x7ff0480406c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7ff04803e200 0x7ff0480406c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff06411d7f0 tx=0x7ff060001c80 comp rx=0 tx=0).stop 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 10 monclient: shutdown 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 1 -- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 msgr2=0x7ff04800b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 1 --2- 172.21.15.37:0/1971004710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0640a3070 0x7ff04800b260 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009f80 tx=0x7ff054008030 comp rx=0 tx=0).stop 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069e57700 1 -- 172.21.15.37:0/1971004710 reap_dead start 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff069e57700 1 -- 172.21.15.37:0/1971004710 reap_dead start 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 1 -- 172.21.15.37:0/1971004710 shutdown_connections 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 1 -- 172.21.15.37:0/1971004710 >> 172.21.15.37:0/1971004710 conn(0x7ff0640a3ea0 msgr2=0x7ff06409f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 1 -- 172.21.15.37:0/1971004710 shutdown_connections 2024-09-16T07:24:59.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.030+0000 7ff06a658700 1 -- 172.21.15.37:0/1971004710 wait complete. 2024-09-16T07:24:59.071 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:24:59.071 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T07:24:59.071 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.190+0000 7f60b0a31700 10 monclient: get_monmap_and_config 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.190+0000 7f60b0a31700 10 monclient: build_initial_monmap 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.190+0000 7f60b0a31700 10 monclient: monmap: 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:24:59.194227+0000 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:24:59.194227+0000 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:24:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:24:59.196 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:24:59.196 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:24:59.196 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:24:59.196 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.190+0000 7f60b0a31700 1 Processor -- start 2024-09-16T07:24:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.190+0000 7f60b0a31700 1 -- start start 2024-09-16T07:24:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.190+0000 7f60b0a31700 10 monclient: init 2024-09-16T07:24:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.190+0000 7f60b0a31700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.190+0000 7f60b0a31700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T07:24:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f60a40b63f0 0x7f60a40234f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 10 monclient(hunting): picked mon.noname-c con 0x7f60a40b63f0 addr v2:172.21.15.143:3300/0 2024-09-16T07:24:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60a40b67d0 addr v1:172.21.15.37:6789/0 2024-09-16T07:24:59.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60a40b6c20 addr v1:172.21.15.143:6789/0 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a40b6030 con 0x7f60a40b67d0 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a4020000 con 0x7f60a40b6c20 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a40ba350 con 0x7f60a40b63f0 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 10 monclient(hunting): authenticate will time out at 1219.680105s 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aa59c700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f60a40b67d0 0x7f60a40b9530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46678/0 (socket says 172.21.15.37:46678) 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aa59c700 1 -- 172.21.15.37:0/3231121621 learned_addr learned my addr 172.21.15.37:0/3231121621 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 1 --2- 172.21.15.37:0/3231121621 >> v2:172.21.15.143:3300/0 conn(0x7f60a40b63f0 0x7f60a40234f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 10 monclient(hunting): get_auth_request con 0x7f60a40b63f0 auth_method 0 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 1 -- 172.21.15.37:0/3231121621 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898921106 0 0) 0x7f60a4020000 con 0x7f60a40b6c20 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient(hunting): _negotiate 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient(hunting): my global_id is 6283 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 1 -- 172.21.15.37:0/3231121621 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60940035d0 con 0x7f60a40b6c20 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 1 -- 172.21.15.37:0/3231121621 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651553656 0 0) 0x7f60a40b6030 con 0x7f60a40b67d0 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient(hunting): _negotiate 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient(hunting): my global_id is 54114 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 1 -- 172.21.15.37:0/3231121621 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60a4020000 con 0x7f60a40b67d0 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 10 monclient(hunting): handle_auth_done global_id 6280 payload 274 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 1 -- 172.21.15.37:0/3231121621 >> v1:172.21.15.143:6789/0 conn(0x7f60a40b6c20 legacy=0x7f60a40b9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 1 -- 172.21.15.37:0/3231121621 >> v1:172.21.15.37:6789/0 conn(0x7f60a40b67d0 legacy=0x7f60a40b9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 10 monclient: _finish_hunting 0 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 1 monclient: found mon.noname-c 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 1 -- 172.21.15.37:0/3231121621 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a401fce0 con 0x7f60a40b63f0 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60aad9d700 1 --2- 172.21.15.37:0/3231121621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60a40b63f0 0x7f60a40234f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009850 tx=0x7f60a0009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a5ceca88bf7f515 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 1 -- 172.21.15.37:0/3231121621 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0011830 con 0x7f60a40b63f0 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient: dump: 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient: _finish_auth 0 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient: _check_auth_tickets 2024-09-16T07:24:59.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 5 monclient: authenticate success, global_id 6280 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 1 -- 172.21.15.37:0/3231121621 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a0011990 con 0x7f60a40b63f0 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 10 monclient: get_monmap_and_config success 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 4 set_mon_vals no callback set 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 1 -- 172.21.15.37:0/3231121621 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0011d40 con 0x7f60a40b63f0 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60a9d9b700 10 monclient: dump: 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.211 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 10 monclient: shutdown 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 -- 172.21.15.37:0/3231121621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60a40b63f0 msgr2=0x7f60a40234f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 --2- 172.21.15.37:0/3231121621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60a40b63f0 0x7f60a40234f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009850 tx=0x7f60a0009b60 comp rx=0 tx=0).stop 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 -- 172.21.15.37:0/3231121621 shutdown_connections 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 --2- 172.21.15.37:0/3231121621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60a40b63f0 0x7f60a40234f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 -- 172.21.15.37:0/3231121621 >> 172.21.15.37:0/3231121621 conn(0x7f60a40064b0 msgr2=0x7f60a40068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:59.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 -- 172.21.15.37:0/3231121621 shutdown_connections 2024-09-16T07:24:59.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 1 -- 172.21.15.37:0/3231121621 wait complete. 2024-09-16T07:24:59.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 10 monclient: build_initial_monmap 2024-09-16T07:24:59.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.194+0000 7f60b0a31700 10 monclient: monmap: 2024-09-16T07:24:59.213 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:59.213 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:24:59.213 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:24:59.213 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 Processor -- start 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 -- start start 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient: init 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60a40b63f0 0x7f60a41505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient(hunting): picked mon.noname-a con 0x7f60a40b63f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:24:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b67d0 0x7f60a40b5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient(hunting): picked mon.noname-c con 0x7f60a40b67d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60a40b6c20 0x7f60a401d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient(hunting): picked mon.noname-b con 0x7f60a40b6c20 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:24:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40a8500 con 0x7f60a40b63f0 2024-09-16T07:24:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40a8750 con 0x7f60a40b6c20 2024-09-16T07:24:59.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a4028c40 con 0x7f60a40b67d0 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient(hunting): authenticate will time out at 1219.685211s 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b67d0 0x7f60a40b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b67d0 0x7f60a40b5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46442/0 (socket says 172.21.15.37:46442) 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 1 -- 172.21.15.37:0/3145251274 learned_addr learned my addr 172.21.15.37:0/3145251274 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 10 monclient(hunting): get_auth_request con 0x7f60a40b67d0 auth_method 0 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aad9d700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60a40b63f0 0x7f60a41505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aad9d700 10 monclient(hunting): get_auth_request con 0x7f60a40b63f0 auth_method 0 2024-09-16T07:24:59.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60ab59e700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60a40b6c20 0x7f60a401d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60ab59e700 10 monclient(hunting): get_auth_request con 0x7f60a40b6c20 auth_method 0 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60ab59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60ab59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60ab59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60ab59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60ab59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60ab59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 10 monclient(hunting): handle_auth_done global_id 44510 payload 824 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 1 -- 172.21.15.37:0/3145251274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60a40b6c20 msgr2=0x7f60a401d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60a40b6c20 0x7f60a401d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 1 -- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60a40b63f0 msgr2=0x7f60a41505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60a40b63f0 0x7f60a41505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 10 monclient: _finish_hunting 0 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 1 monclient: found mon.noname-c 2024-09-16T07:24:59.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 1 -- 172.21.15.37:0/3145251274 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a00094b0 con 0x7f60a40b67d0 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60aa59c700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b67d0 0x7f60a40b5240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f609c00c520 tx=0x7f609c00c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c005d90 con 0x7f60a40b67d0 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 10 monclient: dump: 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.219 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.220 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 10 monclient: _finish_auth 0 2024-09-16T07:24:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 10 monclient: _check_auth_tickets 2024-09-16T07:24:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c0070d0 con 0x7f60a40b67d0 2024-09-16T07:24:59.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 5 monclient: authenticate success, global_id 44510 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c007440 con 0x7f60a40b67d0 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient: _renew_subs 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60a959a700 4 set_mon_vals no callback set 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 -- 172.21.15.37:0/3145251274 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a41512a0 con 0x7f60a40b67d0 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 10 monclient: dump: 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.221 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient: _renew_subs 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 -- 172.21.15.37:0/3145251274 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a40b9dd0 con 0x7f60a40b67d0 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b5780 0x7f60a4001180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60b0a31700 1 -- 172.21.15.37:0/3145251274 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40a88e0 con 0x7f60a40b5780 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.198+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f609c0075a0 con 0x7f60a40b67d0 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b5780 0x7f60a4001180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 10 monclient: get_auth_request con 0x7f60a40b5780 auth_method 0 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 10 monclient(hunting): handle_auth_done global_id 44513 payload 824 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 1 -- 172.21.15.37:0/3145251274 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60a40225c0 con 0x7f60a40b5780 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6098035600 0x7f6098037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f609c046de0 con 0x7f60a40b67d0 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60aad9d700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b5780 0x7f60a4001180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009850 tx=0x7f60a0004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60ab59e700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6098035600 0x7f6098037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0020070 con 0x7f60a40b5780 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 10 monclient: dump: 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.224 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.225 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60ab59e700 10 monclient: get_auth_request con 0x7f6098035600 auth_method 0 2024-09-16T07:24:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f60a0026420 con 0x7f60a40b5780 2024-09-16T07:24:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:24:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b5780 msgr2=0x7f60a4001180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b5780 0x7f60a4001180 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009850 tx=0x7f60a0004770 comp rx=0 tx=0).stop 2024-09-16T07:24:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b5780 msgr2=0x7f60a4001180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:24:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60937fe700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b5780 0x7f60a4001180 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.202+0000 7f60ab59e700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6098035600 0x7f6098037ac0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f60ac068a10 tx=0x7f60ac061dc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.210+0000 7f60b0a31700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:24:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.210+0000 7f60b0a31700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40016c0 0x7f60a4001ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.210+0000 7f60b0a31700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.210+0000 7f60b0a31700 1 -- 172.21.15.37:0/3145251274 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a4007f80 con 0x7f60a40016c0 2024-09-16T07:24:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.210+0000 7f60aad9d700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40016c0 0x7f60a4001ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60aad9d700 10 monclient: get_auth_request con 0x7f60a40016c0 auth_method 0 2024-09-16T07:24:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60aad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60aad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60aad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60aad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60aad9d700 10 monclient(hunting): handle_auth_done global_id 44516 payload 824 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60aad9d700 1 -- 172.21.15.37:0/3145251274 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f60a401c960 con 0x7f60a40016c0 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60aad9d700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40016c0 0x7f60a4001ad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f60a0000c00 tx=0x7f60a0011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a002e6d0 con 0x7f60a40016c0 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 10 monclient: dump: 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4619 (secure 0 0 0) 0x7f60a002e830 con 0x7f60a40016c0 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40016c0 msgr2=0x7f60a4001ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40016c0 0x7f60a4001ad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f60a0000c00 tx=0x7f60a0011000 comp rx=0 tx=0).stop 2024-09-16T07:24:59.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 1 -- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40016c0 msgr2=0x7f60a4001ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60937fe700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40016c0 0x7f60a4001ad0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 28, 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:24:59.230 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:24:59.231 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:24:59.232 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:24:59.233 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:24:59.234 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:24:59.235 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.236 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:24:59.237 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:24:59.238 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.239 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:24:59.240 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.241 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.242 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.243 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.243 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:24:59.243 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:24:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60b0a31700 1 -- 172.21.15.37:0/3145251274 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6098035600 msgr2=0x7f6098037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60b0a31700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6098035600 0x7f6098037ac0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f60ac068a10 tx=0x7f60ac061dc0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60b0a31700 10 monclient: shutdown 2024-09-16T07:24:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60b0a31700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60b0a31700 1 -- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b67d0 msgr2=0x7f60a40b5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60b0a31700 1 --2- 172.21.15.37:0/3145251274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60a40b67d0 0x7f60a40b5240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f609c00c520 tx=0x7f609c00c830 comp rx=0 tx=0).stop 2024-09-16T07:24:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60ab59e700 1 -- 172.21.15.37:0/3145251274 reap_dead start 2024-09-16T07:24:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60ab59e700 1 -- 172.21.15.37:0/3145251274 reap_dead start 2024-09-16T07:24:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60b0a31700 1 -- 172.21.15.37:0/3145251274 shutdown_connections 2024-09-16T07:24:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60b0a31700 1 -- 172.21.15.37:0/3145251274 >> 172.21.15.37:0/3145251274 conn(0x7f60a40064b0 msgr2=0x7f60a40b86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60b0a31700 1 -- 172.21.15.37:0/3145251274 shutdown_connections 2024-09-16T07:24:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.214+0000 7f60b0a31700 1 -- 172.21.15.37:0/3145251274 wait complete. 2024-09-16T07:24:59.245 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:24:59.245 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T07:24:59.245 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T07:24:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient: get_monmap_and_config 2024-09-16T07:24:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient: build_initial_monmap 2024-09-16T07:24:59.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient: monmap: 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:24:59.369895+0000 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:24:59.369895+0000 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:24:59.371 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 1 Processor -- start 2024-09-16T07:24:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 1 -- start start 2024-09-16T07:24:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient: init 2024-09-16T07:24:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T07:24:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f368c1334e0 addr v1:172.21.15.37:6789/0 2024-09-16T07:24:59.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f368c09cd00 0x7f368c1383d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient(hunting): picked mon.noname-c con 0x7f368c09cd00 addr v2:172.21.15.143:3300/0 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f368c138910 addr v1:172.21.15.143:6789/0 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f368c09a030 con 0x7f368c1334e0 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f368c133fd0 con 0x7f368c138910 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368c071200 con 0x7f368c09cd00 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.366+0000 7f3693762700 10 monclient(hunting): authenticate will time out at 1219.855813s 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f36914fe700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f368c1334e0 0x7f368c1338c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46682/0 (socket says 172.21.15.37:46682) 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f36914fe700 1 -- 172.21.15.37:0/2384694352 learned_addr learned my addr 172.21.15.37:0/2384694352 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 1 --2- 172.21.15.37:0/2384694352 >> v2:172.21.15.143:3300/0 conn(0x7f368c09cd00 0x7f368c1383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 10 monclient(hunting): get_auth_request con 0x7f368c09cd00 auth_method 0 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 1 -- 172.21.15.37:0/2384694352 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507614623 0 0) 0x7f368c09a030 con 0x7f368c1334e0 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient(hunting): _negotiate 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient(hunting): my global_id is 54120 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 1 -- 172.21.15.37:0/2384694352 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36780035d0 con 0x7f368c1334e0 2024-09-16T07:24:59.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 10 monclient(hunting): handle_auth_done global_id 6289 payload 274 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 1 -- 172.21.15.37:0/2384694352 >> v1:172.21.15.143:6789/0 conn(0x7f368c138910 legacy=0x7f368c13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 1 -- 172.21.15.37:0/2384694352 >> v1:172.21.15.37:6789/0 conn(0x7f368c1334e0 legacy=0x7f368c1338c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 10 monclient: _finish_hunting 0 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 1 monclient: found mon.noname-c 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 1 -- 172.21.15.37:0/2384694352 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c09a230 con 0x7f368c09cd00 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3690cfd700 1 --2- 172.21.15.37:0/2384694352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c09cd00 0x7f368c1383d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f367c009850 tx=0x7f367c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f110a3133245a5e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 1 -- 172.21.15.37:0/2384694352 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c011830 con 0x7f368c09cd00 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient: dump: 2024-09-16T07:24:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient: _finish_auth 0 2024-09-16T07:24:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient: _check_auth_tickets 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 5 monclient: authenticate success, global_id 6289 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 1 -- 172.21.15.37:0/2384694352 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367c011990 con 0x7f368c09cd00 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 10 monclient: get_monmap_and_config success 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 4 set_mon_vals no callback set 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 1 -- 172.21.15.37:0/2384694352 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c011d40 con 0x7f368c09cd00 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:24:59.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3683fff700 10 monclient: dump: 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 10 monclient: shutdown 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:59.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 1 -- 172.21.15.37:0/2384694352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c09cd00 msgr2=0x7f368c1383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 1 --2- 172.21.15.37:0/2384694352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c09cd00 0x7f368c1383d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f367c009850 tx=0x7f367c009b60 comp rx=0 tx=0).stop 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 1 -- 172.21.15.37:0/2384694352 shutdown_connections 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 1 --2- 172.21.15.37:0/2384694352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c09cd00 0x7f368c1383d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 1 -- 172.21.15.37:0/2384694352 >> 172.21.15.37:0/2384694352 conn(0x7f368c0a04b0 msgr2=0x7f368c0a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 1 -- 172.21.15.37:0/2384694352 shutdown_connections 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 1 -- 172.21.15.37:0/2384694352 wait complete. 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 10 monclient: build_initial_monmap 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.370+0000 7f3693762700 10 monclient: monmap: 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:59.383 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:24:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:24:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:24:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:24:59.384 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 Processor -- start 2024-09-16T07:24:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 -- start start 2024-09-16T07:24:59.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient: init 2024-09-16T07:24:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:24:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f368c09cd00 0x7f368c1c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient(hunting): picked mon.noname-a con 0x7f368c09cd00 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:24:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f368c1334e0 0x7f368c1c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient(hunting): picked mon.noname-c con 0x7f368c1334e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c138910 0x7f368c1c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient(hunting): picked mon.noname-b con 0x7f368c138910 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:24:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c11b250 con 0x7f368c09cd00 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c11b730 con 0x7f368c138910 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368c09f580 con 0x7f368c1334e0 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient(hunting): authenticate will time out at 1219.861073s 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3690cfd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f368c1334e0 0x7f368c1c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f368c09cd00 0x7f368c1c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3690cfd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f368c1334e0 0x7f368c1c8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46476/0 (socket says 172.21.15.37:46476) 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3690cfd700 1 -- 172.21.15.37:0/3845798789 learned_addr learned my addr 172.21.15.37:0/3845798789 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3690cfd700 10 monclient(hunting): get_auth_request con 0x7f368c1334e0 auth_method 0 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3690cfd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3690cfd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3690cfd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 10 monclient(hunting): get_auth_request con 0x7f368c09cd00 auth_method 0 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3691cff700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c138910 0x7f368c1c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3691cff700 10 monclient(hunting): get_auth_request con 0x7f368c138910 auth_method 0 2024-09-16T07:24:59.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3691cff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3691cff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3691cff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 10 monclient(hunting): handle_auth_done global_id 54123 payload 824 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 1 -- 172.21.15.37:0/3845798789 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f368c1334e0 msgr2=0x7f368c1c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f368c1334e0 0x7f368c1c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 1 -- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c138910 msgr2=0x7f368c1c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c138910 0x7f368c1c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 10 monclient: _finish_hunting 0 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 1 monclient: found mon.noname-a 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 1 -- 172.21.15.37:0/3845798789 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c0094b0 con 0x7f368c09cd00 2024-09-16T07:24:59.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3691cff700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c138910 0x7f368c1c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36914fe700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f368c09cd00 0x7f368c1c8230 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f368800c500 tx=0x7f368800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688005d90 con 0x7f368c09cd00 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 10 monclient: dump: 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 10 monclient: _finish_auth 0 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 5 monclient: authenticate success, global_id 54123 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36880070d0 con 0x7f368c09cd00 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient: _renew_subs 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 -- 172.21.15.37:0/3845798789 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368c138250 con 0x7f368c09cd00 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688007440 con 0x7f368c09cd00 2024-09-16T07:24:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f36837fe700 4 set_mon_vals no callback set 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3681ffb700 10 monclient: dump: 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.396 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient: _renew_subs 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 -- 172.21.15.37:0/3845798789 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368c11b450 con 0x7f368c09cd00 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136190 0x7f368c136570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.374+0000 7f3693762700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3693762700 1 -- 172.21.15.37:0/3845798789 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c0675e0 con 0x7f368c136190 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f36880075a0 con 0x7f368c09cd00 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136190 0x7f368c136570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 10 monclient: get_auth_request con 0x7f368c136190 auth_method 0 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3670035630 0x7f3670037af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f3688046eb0 con 0x7f368c09cd00 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 10 monclient(hunting): handle_auth_done global_id 6298 payload 824 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 1 -- 172.21.15.37:0/3845798789 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f368c06c750 con 0x7f368c136190 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3691cff700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3670035630 0x7f3670037af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3691cff700 10 monclient: get_auth_request con 0x7f3670035630 auth_method 0 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3690cfd700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136190 0x7f368c136570 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f368c1328b0 tx=0x7f367c004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c020070 con 0x7f368c136190 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 10 monclient: dump: 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3691cff700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3670035630 0x7f3670037af0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f368c11adc0 tx=0x7f3684009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+9332 (secure 0 0 0) 0x7f367c0117d0 con 0x7f368c136190 2024-09-16T07:24:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:24:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136190 msgr2=0x7f368c136570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136190 0x7f368c136570 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f368c1328b0 tx=0x7f367c004770 comp rx=0 tx=0).stop 2024-09-16T07:24:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136190 msgr2=0x7f368c136570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:24:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.378+0000 7f3681ffb700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136190 0x7f368c136570 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3693762700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3693762700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136ab0 0x7f368c06e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3693762700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3693762700 1 -- 172.21.15.37:0/3845798789 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c1338c0 con 0x7f368c136ab0 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3690cfd700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136ab0 0x7f368c06e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3690cfd700 10 monclient: get_auth_request con 0x7f368c136ab0 auth_method 0 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3690cfd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3690cfd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3690cfd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.390+0000 7f3690cfd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3690cfd700 10 monclient(hunting): handle_auth_done global_id 6301 payload 824 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3690cfd700 1 -- 172.21.15.37:0/3845798789 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f368c070870 con 0x7f368c136ab0 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3690cfd700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136ab0 0x7f368c06e310 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f368c09f760 tx=0x7f367c011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c01be80 con 0x7f368c136ab0 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 10 monclient: dump: 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 <== mon.1 v2:172.21.15.143:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4787 (secure 0 0 0) 0x7f367c031070 con 0x7f368c136ab0 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136ab0 msgr2=0x7f368c06e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136ab0 0x7f368c06e310 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f368c09f760 tx=0x7f367c011000 comp rx=0 tx=0).stop 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 1 -- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136ab0 msgr2=0x7f368c06e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T07:24:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3681ffb700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f368c136ab0 0x7f368c06e310 secure :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f368c09f760 tx=0x7f367c011000 comp rx=0 tx=0).stop 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 28, 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-09-16T07:24:59.405 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:24:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2449958758054445060", 2024-09-16T07:24:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-09-16T07:24:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:24:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:24:59.406 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:24:59.407 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540138322906710015", 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:24:59.408 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-09-16T07:24:59.409 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "2b3c1781-6e4a-42f3-a6cb-6f1008258949", 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-09-16T07:18:06.498847Z", 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 18, 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "reef", 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-09-16T07:24:59.410 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "reef" 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.411 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:24:59.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-09-16T07:24:59.413 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:3300", 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789", 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.414 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.143:6789/0", 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.143:6789/0", 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.415 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-09-16T07:24:59.416 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-09-16T07:24:59.417 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-09-16T07:24:59.418 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-09-16T07:24:59.419 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-16T07:24:59.419 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f01cfbffffdffff", 2024-09-16T07:24:59.419 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "reef", 2024-09-16T07:24:59.419 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-09-16T07:24:59.419 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-16T07:24:59.419 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-16T07:24:59.419 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-16T07:24:59.419 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-09-16T07:24:59.419 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 1 -- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3670035630 msgr2=0x7f3670037af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3670035630 0x7f3670037af0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f368c11adc0 tx=0x7f3684009040 comp rx=0 tx=0).stop 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 10 monclient: shutdown 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 1 -- 172.21.15.37:0/3845798789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f368c09cd00 msgr2=0x7f368c1c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f368c09cd00 0x7f368c1c8230 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f368800c500 tx=0x7f368800c810 comp rx=0 tx=0).stop 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3691cff700 1 -- 172.21.15.37:0/3845798789 reap_dead start 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 1 -- 172.21.15.37:0/3845798789 shutdown_connections 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 1 --2- 172.21.15.37:0/3845798789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f368c09cd00 0x7f368c1c8230 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 1 -- 172.21.15.37:0/3845798789 >> 172.21.15.37:0/3845798789 conn(0x7f368c0a04b0 msgr2=0x7f368c09e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 1 -- 172.21.15.37:0/3845798789 shutdown_connections 2024-09-16T07:24:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.394+0000 7f3693762700 1 -- 172.21.15.37:0/3845798789 wait complete. 2024-09-16T07:24:59.422 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T07:24:59.422 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T07:24:59.422 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T07:24:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient: get_monmap_and_config 2024-09-16T07:24:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient: build_initial_monmap 2024-09-16T07:24:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient: monmap: 2024-09-16T07:24:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:24:59.544006+0000 2024-09-16T07:24:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:24:59.544006+0000 2024-09-16T07:24:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:59.545 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:24:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:24:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:24:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:24:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:24:59.546 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 1 Processor -- start 2024-09-16T07:24:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 1 -- start start 2024-09-16T07:24:59.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient: init 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa24c0a2750 addr v1:172.21.15.143:6789/0 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa24c0a3240 addr v1:172.21.15.37:6789/0 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c139a10 0x7fa24c13be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa24c139a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24c09d810 con 0x7fa24c0a3240 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24c099be0 con 0x7fa24c0a2750 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c09de40 con 0x7fa24c139a10 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient(hunting): authenticate will time out at 1220.029974s 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa249d9b700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa24c0a3240 0x7fa24c1382f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46692/0 (socket says 172.21.15.37:46692) 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa249d9b700 1 -- 172.21.15.37:0/988748057 learned_addr learned my addr 172.21.15.37:0/988748057 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 1 --2- 172.21.15.37:0/988748057 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c139a10 0x7fa24c13be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 10 monclient(hunting): get_auth_request con 0x7fa24c139a10 auth_method 0 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 10 monclient(hunting): handle_auth_done global_id 44522 payload 274 2024-09-16T07:24:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 1 -- 172.21.15.37:0/988748057 >> v1:172.21.15.143:6789/0 conn(0x7fa24c0a2750 legacy=0x7fa24c0a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 1 -- 172.21.15.37:0/988748057 >> v1:172.21.15.37:6789/0 conn(0x7fa24c0a3240 legacy=0x7fa24c1382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 1 -- 172.21.15.37:0/988748057 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265622257 0 0) 0x7fa24c09d810 con 0x7fa24c0a3240 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 10 monclient: _finish_hunting 0 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 1 monclient: found mon.noname-b 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 1 -- 172.21.15.37:0/988748057 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c09d530 con 0x7fa24c139a10 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24ad9d700 1 --2- 172.21.15.37:0/988748057 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c139a10 0x7fa24c13be00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa244009dc0 tx=0x7fa24400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16204aa16b868f88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 1 -- 172.21.15.37:0/988748057 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24400e070 con 0x7fa24c139a10 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 10 monclient: dump: 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 10 monclient: _finish_auth 0 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 10 monclient: _check_auth_tickets 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 5 monclient: authenticate success, global_id 44522 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 1 -- 172.21.15.37:0/988748057 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24400bbd0 con 0x7fa24c139a10 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 10 monclient: get_monmap_and_config success 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24b59e700 4 set_mon_vals no callback set 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 1 -- 172.21.15.37:0/988748057 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244012430 con 0x7fa24c139a10 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.542+0000 7fa24959a700 10 monclient: dump: 2024-09-16T07:24:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient: shutdown 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 -- 172.21.15.37:0/988748057 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c139a10 msgr2=0x7fa24c13be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 --2- 172.21.15.37:0/988748057 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c139a10 0x7fa24c13be00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa244009dc0 tx=0x7fa24400b330 comp rx=0 tx=0).stop 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 -- 172.21.15.37:0/988748057 shutdown_connections 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 --2- 172.21.15.37:0/988748057 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c139a10 0x7fa24c13be00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 -- 172.21.15.37:0/988748057 >> 172.21.15.37:0/988748057 conn(0x7fa24c0a3ea0 msgr2=0x7fa24c0a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 -- 172.21.15.37:0/988748057 shutdown_connections 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 -- 172.21.15.37:0/988748057 wait complete. 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient: build_initial_monmap 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient: monmap: 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:24:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:24:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:24:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:24:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:24:59.557 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 Processor -- start 2024-09-16T07:24:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 -- start start 2024-09-16T07:24:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient: init 2024-09-16T07:24:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient: _reopen_session rank -1 2024-09-16T07:24:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c0a2750 0x7fa24c1c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa24c0a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24c0a3240 0x7fa24c137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa24c0a3240 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c139a10 0x7fa24c1ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa24c139a10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c124f80 con 0x7fa24c139a10 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c06fc30 con 0x7fa24c0a3240 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c124ca0 con 0x7fa24c0a2750 2024-09-16T07:24:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient(hunting): _renew_subs 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24b59e700 10 monclient(hunting): authenticate will time out at 1220.035112s 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24ad9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c139a10 0x7fa24c1ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24a59c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c0a2750 0x7fa24c1c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24ad9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c139a10 0x7fa24c1ca130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:38072/0 (socket says 172.21.15.37:38072) 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24a59c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c0a2750 0x7fa24c1c6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46502/0 (socket says 172.21.15.37:46502) 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24ad9d700 1 -- 172.21.15.37:0/4058659964 learned_addr learned my addr 172.21.15.37:0/4058659964 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24ad9d700 10 monclient(hunting): get_auth_request con 0x7fa24c139a10 auth_method 0 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24a59c700 10 monclient(hunting): get_auth_request con 0x7fa24c0a2750 auth_method 0 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24a59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa249d9b700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24c0a3240 0x7fa24c137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa249d9b700 10 monclient(hunting): get_auth_request con 0x7fa24c0a3240 auth_method 0 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa249d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa249d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa249d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa24a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa249d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa249d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:24:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.546+0000 7fa249d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 10 monclient(hunting): handle_auth_done global_id 54129 payload 824 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 1 -- 172.21.15.37:0/4058659964 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c0a2750 msgr2=0x7fa24c1c6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c0a2750 0x7fa24c1c6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 1 -- 172.21.15.37:0/4058659964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24c0a3240 msgr2=0x7fa24c137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24c0a3240 0x7fa24c137b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 10 monclient: _finish_hunting 0 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 1 monclient: found mon.noname-a 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 1 -- 172.21.15.37:0/4058659964 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa244009a70 con 0x7fa24c139a10 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24a59c700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c0a2750 0x7fa24c1c6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24ad9d700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c139a10 0x7fa24c1ca130 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa244014040 tx=0x7fa244004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 1 -- 172.21.15.37:0/4058659964 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24401b8d0 con 0x7fa24c139a10 2024-09-16T07:24:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 10 monclient: dump: 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 10 monclient: _finish_auth 0 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 10 monclient: _check_auth_tickets 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 1 -- 172.21.15.37:0/4058659964 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa244020070 con 0x7fa24c139a10 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 5 monclient: authenticate success, global_id 54129 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 10 monclient: _renew_subs 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 1 -- 172.21.15.37:0/4058659964 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24c0724c0 con 0x7fa24c139a10 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 1 -- 172.21.15.37:0/4058659964 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24401bcb0 con 0x7fa24c139a10 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 10 monclient: dump: 2024-09-16T07:24:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:24:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:24:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:24:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:24:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:24:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:24:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:24:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:24:59.568 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:24:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 10 monclient: _renew_subs 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 1 -- 172.21.15.37:0/4058659964 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24c125540 con 0x7fa24c139a10 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa248d99700 4 set_mon_vals no callback set 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 1 -- 172.21.15.37:0/4058659964 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fa24401c400 con 0x7fa24c139a10 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24b59e700 1 -- 172.21.15.37:0/4058659964 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24c122aa0 con 0x7fa24c139a10 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa22c035600 0x7fa22c037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa23affd700 1 -- 172.21.15.37:0/4058659964 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fa244022070 con 0x7fa24c139a10 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24a59c700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa22c035600 0x7fa22c037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24a59c700 10 monclient: get_auth_request con 0x7fa22c035600 auth_method 0 2024-09-16T07:24:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.550+0000 7fa24a59c700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa22c035600 0x7fa22c037ac0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa240000f60 tx=0x7fa240009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:24:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.558+0000 7fa23affd700 1 -- 172.21.15.37:0/4058659964 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa24c122aa0 con 0x7fa24c139a10 2024-09-16T07:24:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.558+0000 7fa23affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:24:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.558+0000 7fa23affd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:24:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.790+0000 7fa24b59e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T07:24:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.790+0000 7fa24b59e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T07:24:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.790+0000 7fa24b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:24:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.790+0000 7fa24b59e700 1 -- 172.21.15.37:0/4058659964 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fa24c1250c0 con 0x7fa24c139a10 2024-09-16T07:24:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.790+0000 7fa23affd700 1 -- 172.21.15.37:0/4058659964 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fa24c1250c0 con 0x7fa24c139a10 2024-09-16T07:24:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.790+0000 7fa23affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T07:24:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.790+0000 7fa23affd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:24:59.796 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.795+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.796 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.795+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=12,kv=12,logm=62} crc {auth=3207563495,config=1802735181,health=1535303502,kv=3696321355,logm=279112970}) 2024-09-16T07:24:59.796 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.795+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=12,logm=63} crc {auth=3207563495,config=3541143443,health=1791238353,kv=3696321355,logm=180042388}) 2024-09-16T07:24:59.796 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.795+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.796 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.795+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=12,kv=12,logm=62} crc {auth=3207563495,config=1802735181,health=1535303502,kv=3696321355,logm=279112970}) 2024-09-16T07:24:59.796 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.795+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3207563495,config=3541143443,health=1096256309,kv=301621784,logm=3328394003}) 2024-09-16T07:24:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 -- 172.21.15.37:0/4058659964 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa22c035600 msgr2=0x7fa22c037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa22c035600 0x7fa22c037ac0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa240000f60 tx=0x7fa240009040 comp rx=0 tx=0).stop 2024-09-16T07:24:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 10 monclient: shutdown 2024-09-16T07:24:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:24:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 -- 172.21.15.37:0/4058659964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c139a10 msgr2=0x7fa24c1ca130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:24:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c139a10 0x7fa24c1ca130 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa244014040 tx=0x7fa244004af0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 -- 172.21.15.37:0/4058659964 shutdown_connections 2024-09-16T07:24:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c139a10 0x7fa24c1ca130 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24c0a3240 0x7fa24c137b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa22c035600 0x7fa22c037ac0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 --2- 172.21.15.37:0/4058659964 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c0a2750 0x7fa24c1c6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:24:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 -- 172.21.15.37:0/4058659964 >> 172.21.15.37:0/4058659964 conn(0x7fa24c0a3ea0 msgr2=0x7fa24c138770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:24:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 -- 172.21.15.37:0/4058659964 shutdown_connections 2024-09-16T07:24:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:24:59.794+0000 7fa24b59e700 1 -- 172.21.15.37:0/4058659964 wait complete. 2024-09-16T07:24:59.800 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.799+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.800 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.799+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2000629301}) 2024-09-16T07:24:59.800 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.799+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833815827}) 2024-09-16T07:24:59.800 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.799+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.800 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.799+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2000629301}) 2024-09-16T07:24:59.800 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.799+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1547022069}) 2024-09-16T07:24:59.801 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.801 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434662534}) 2024-09-16T07:24:59.801 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123613231}) 2024-09-16T07:24:59.801 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.801 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434662534}) 2024-09-16T07:24:59.801 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226538794}) 2024-09-16T07:24:59.803 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.803 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=718620474}) 2024-09-16T07:24:59.803 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3441896906}) 2024-09-16T07:24:59.803 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.803 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=718620474}) 2024-09-16T07:24:59.803 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.803+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412619882}) 2024-09-16T07:24:59.805 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.805 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417852054}) 2024-09-16T07:24:59.805 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426508375}) 2024-09-16T07:24:59.805 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.806 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3417852054}) 2024-09-16T07:24:59.806 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3299795414}) 2024-09-16T07:24:59.807 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.807 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3348429476}) 2024-09-16T07:24:59.807 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555123145}) 2024-09-16T07:24:59.807 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.807 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3348429476}) 2024-09-16T07:24:59.807 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.807+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526889028}) 2024-09-16T07:24:59.809 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.809 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3352840302}) 2024-09-16T07:24:59.809 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680034658}) 2024-09-16T07:24:59.809 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.809 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3352840302}) 2024-09-16T07:24:59.809 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1111041314}) 2024-09-16T07:24:59.811 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T07:24:59.811 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.811 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229833842}) 2024-09-16T07:24:59.811 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1091836617}) 2024-09-16T07:24:59.812 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.812 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229833842}) 2024-09-16T07:24:59.812 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.811+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1826549949}) 2024-09-16T07:24:59.812 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.812 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=932992425}) 2024-09-16T07:24:59.812 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1441675088}) 2024-09-16T07:24:59.813 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.813 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=932992425}) 2024-09-16T07:24:59.813 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195008974}) 2024-09-16T07:24:59.814 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.814 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3665776094}) 2024-09-16T07:24:59.814 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249469378}) 2024-09-16T07:24:59.814 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.814 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3665776094}) 2024-09-16T07:24:59.814 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=935933976}) 2024-09-16T07:24:59.815 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.815 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2285580948}) 2024-09-16T07:24:59.815 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689599240}) 2024-09-16T07:24:59.815 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.815 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2285580948}) 2024-09-16T07:24:59.815 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.815+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1509797829}) 2024-09-16T07:24:59.816 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.816 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1363137257}) 2024-09-16T07:24:59.816 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=618123270}) 2024-09-16T07:24:59.816 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.816 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1363137257}) 2024-09-16T07:24:59.816 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3047863788}) 2024-09-16T07:24:59.817 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.817 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2827183459}) 2024-09-16T07:24:59.817 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1744996053}) 2024-09-16T07:24:59.817 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.817 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2827183459}) 2024-09-16T07:24:59.817 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1947453386}) 2024-09-16T07:24:59.818 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.818 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=68,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=8} crc {logm=1836604181,mdsmap=1531151339,mgr=2249125188,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=415186423,mon_config_key=2963502920}) 2024-09-16T07:24:59.818 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=62,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=2} crc {logm=3092527910,mdsmap=1531151339,mgr=3395767200,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=3664676338,mon_config_key=1054993855,monmap=38151330}) 2024-09-16T07:24:59.818 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.819 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=68,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=8} crc {logm=1836604181,mdsmap=1531151339,mgr=2249125188,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=415186423,mon_config_key=2963502920}) 2024-09-16T07:24:59.819 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.819+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12} crc {logm=1180375713,mdsmap=1531151339,mgr=49779396,mgr_command_descs=794974419,mgr_metadata=3661887347,mgrstat=4269559017,mon_config_key=1837016248}) 2024-09-16T07:24:59.819 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.819 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=6,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=82} crc {mon_config_key=2311449113,monmap=1839276543,osd_metadata=598972236,osd_pg_creating=3755685822,osd_snap=1614000649,osdmap=1984816367}) 2024-09-16T07:24:59.820 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=83} crc {mon_config_key=1468921699,monmap=1839276543,osd_metadata=598972236,osd_pg_creating=3755685822,osd_snap=1614000649,osdmap=168762052}) 2024-09-16T07:24:59.820 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.820 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=6,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=82} crc {mon_config_key=2311449113,monmap=1839276543,osd_metadata=598972236,osd_pg_creating=3755685822,osd_snap=1614000649,osdmap=1984816367}) 2024-09-16T07:24:59.820 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=84} crc {mon_config_key=1468921699,monmap=1839276543,osd_metadata=814008949,osd_pg_creating=3755685822,osd_snap=3823785925,osdmap=521222703}) 2024-09-16T07:24:59.821 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.821 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2309539634}) 2024-09-16T07:24:59.821 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1577540897}) 2024-09-16T07:24:59.821 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.821 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2309539634}) 2024-09-16T07:24:59.821 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3179187844}) 2024-09-16T07:24:59.822 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.822 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3497049878}) 2024-09-16T07:24:59.822 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3956213690}) 2024-09-16T07:24:59.822 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.822 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3497049878}) 2024-09-16T07:24:59.822 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=375769398}) 2024-09-16T07:24:59.823 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.823 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3224139993}) 2024-09-16T07:24:59.823 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3295396175}) 2024-09-16T07:24:59.823 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.824 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3224139993}) 2024-09-16T07:24:59.824 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.823+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2591945470}) 2024-09-16T07:24:59.825 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.827+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.825 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.827+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=81} crc {osdmap=3559715830}) 2024-09-16T07:24:59.825 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.827+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=1392392211}) 2024-09-16T07:24:59.825 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.827+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T07:24:59.825 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.827+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=81} crc {osdmap=3559715830}) 2024-09-16T07:24:59.825 INFO:tasks.ceph.mon.b.smithi143.stderr:2024-09-16T07:24:59.827+0000 7fe2dfbb1700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=78} crc {osdmap=1330811309}) 2024-09-16T07:25:00.812 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T07:25:00.813 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T07:25:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.926+0000 7f4d53fff700 10 monclient: get_monmap_and_config 2024-09-16T07:25:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.926+0000 7f4d53fff700 10 monclient: build_initial_monmap 2024-09-16T07:25:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient: monmap: 2024-09-16T07:25:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:00.931403+0000 2024-09-16T07:25:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:00.931403+0000 2024-09-16T07:25:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:00.933 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:00.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 1 Processor -- start 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 1 -- start start 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient: init 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d540a2750 0x7f4d5409d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient(hunting): picked mon.noname-b con 0x7f4d540a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4d540a2b30 0x7f4d540a2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient(hunting): picked mon.noname-c con 0x7f4d540a2b30 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d54132010 addr v1:172.21.15.143:6789/0 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d5409dac0 con 0x7f4d54132010 2024-09-16T07:25:00.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d5409c6c0 con 0x7f4d540a2b30 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5409c9a0 con 0x7f4d540a2750 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient(hunting): authenticate will time out at 1221.417366s 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d540a2750 0x7f4d5409d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d540a2750 0x7f4d5409d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34906/0 (socket says 172.21.15.37:34906) 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 1 -- 172.21.15.37:0/373212938 learned_addr learned my addr 172.21.15.37:0/373212938 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 10 monclient(hunting): get_auth_request con 0x7f4d540a2750 auth_method 0 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:00.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d4a7fc700 1 --2- 172.21.15.37:0/373212938 >> v2:172.21.15.143:3300/0 conn(0x7f4d540a2b30 0x7f4d540a2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d4a7fc700 10 monclient(hunting): get_auth_request con 0x7f4d540a2b30 auth_method 0 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d4a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d4a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d4a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d4a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d4a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d4a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 10 monclient(hunting): handle_auth_done global_id 44528 payload 274 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 1 -- 172.21.15.37:0/373212938 >> v2:172.21.15.143:3300/0 conn(0x7f4d540a2b30 msgr2=0x7f4d540a2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:00.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 1 --2- 172.21.15.37:0/373212938 >> v2:172.21.15.143:3300/0 conn(0x7f4d540a2b30 0x7f4d540a2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 1 -- 172.21.15.37:0/373212938 >> v1:172.21.15.143:6789/0 conn(0x7f4d54132010 legacy=0x7f4d541324a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 10 monclient: _finish_hunting 0 2024-09-16T07:25:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 1 monclient: found mon.noname-b 2024-09-16T07:25:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:25:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 1 -- 172.21.15.37:0/373212938 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5409b130 con 0x7f4d540a2750 2024-09-16T07:25:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d52ffd700 1 --2- 172.21.15.37:0/373212938 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d540a2750 0x7f4d5409d580 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009910 tx=0x7f4d44009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1310d572a0179fce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 1 -- 172.21.15.37:0/373212938 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d44011830 con 0x7f4d540a2750 2024-09-16T07:25:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:00.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 10 monclient: dump: 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:00.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 10 monclient: _finish_auth 0 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 10 monclient: _check_auth_tickets 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 5 monclient: authenticate success, global_id 44528 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 1 -- 172.21.15.37:0/373212938 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d44011990 con 0x7f4d540a2750 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d53fff700 4 set_mon_vals no callback set 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 1 -- 172.21.15.37:0/373212938 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d44011d40 con 0x7f4d540a2750 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.930+0000 7f4d527fc700 10 monclient: dump: 2024-09-16T07:25:00.948 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient: shutdown 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 -- 172.21.15.37:0/373212938 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d540a2750 msgr2=0x7f4d5409d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:00.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 --2- 172.21.15.37:0/373212938 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d540a2750 0x7f4d5409d580 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009910 tx=0x7f4d44009c20 comp rx=0 tx=0).stop 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 -- 172.21.15.37:0/373212938 shutdown_connections 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 --2- 172.21.15.37:0/373212938 >> v2:172.21.15.143:3300/0 conn(0x7f4d540a2b30 0x7f4d540a2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 --2- 172.21.15.37:0/373212938 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d540a2750 0x7f4d5409d580 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 -- 172.21.15.37:0/373212938 >> 172.21.15.37:0/373212938 conn(0x7f4d540a3ea0 msgr2=0x7f4d540a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 -- 172.21.15.37:0/373212938 shutdown_connections 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 -- 172.21.15.37:0/373212938 wait complete. 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient: build_initial_monmap 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient: monmap: 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:00.950 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 Processor -- start 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 -- start start 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient: init 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d540a2750 0x7f4d5413a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient(hunting): picked mon.noname-b con 0x7f4d540a2750 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:00.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d540a2b30 0x7f4d5412d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient(hunting): picked mon.noname-a con 0x7f4d540a2b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d54132010 0x7f4d5412da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient(hunting): picked mon.noname-c con 0x7f4d54132010 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5409ec60 con 0x7f4d540a2b30 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d540724c0 con 0x7f4d540a2750 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d54130b50 con 0x7f4d54132010 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d53fff700 10 monclient(hunting): authenticate will time out at 1221.422832s 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d540a2b30 0x7f4d5412d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:00.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d540a2b30 0x7f4d5412d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39294/0 (socket says 172.21.15.37:39294) 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 1 -- 172.21.15.37:0/2328515632 learned_addr learned my addr 172.21.15.37:0/2328515632 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 10 monclient(hunting): get_auth_request con 0x7f4d540a2b30 auth_method 0 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d537fe700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d54132010 0x7f4d5412da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d52ffd700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d540a2750 0x7f4d5413a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d537fe700 10 monclient(hunting): get_auth_request con 0x7f4d54132010 auth_method 0 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d537fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:00.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d52ffd700 10 monclient(hunting): get_auth_request con 0x7f4d540a2750 auth_method 0 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d52ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d52ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d52ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d52ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 10 monclient(hunting): handle_auth_done global_id 54132 payload 824 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 1 -- 172.21.15.37:0/2328515632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d54132010 msgr2=0x7f4d5412da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:00.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d54132010 0x7f4d5412da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 1 -- 172.21.15.37:0/2328515632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d540a2750 msgr2=0x7f4d5413a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d540a2750 0x7f4d5413a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 10 monclient: _finish_hunting 0 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 1 monclient: found mon.noname-a 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.934+0000 7f4d4a7fc700 1 -- 172.21.15.37:0/2328515632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d44009570 con 0x7f4d540a2b30 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4a7fc700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d540a2b30 0x7f4d5412d4c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00e9d0 tx=0x7f4d3c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 1 -- 172.21.15.37:0/2328515632 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c009720 con 0x7f4d540a2b30 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:00.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 10 monclient: dump: 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 10 monclient: _finish_auth 0 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 10 monclient: _check_auth_tickets 2024-09-16T07:25:00.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 5 monclient: authenticate success, global_id 54132 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 10 monclient: _renew_subs 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 1 -- 172.21.15.37:0/2328515632 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c00ee70 con 0x7f4d540a2b30 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 1 -- 172.21.15.37:0/2328515632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d54130e30 con 0x7f4d540a2b30 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 1 -- 172.21.15.37:0/2328515632 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c00ca50 con 0x7f4d540a2b30 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d51ffb700 4 set_mon_vals no callback set 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 10 monclient: dump: 2024-09-16T07:25:00.957 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 10 monclient: _renew_subs 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 1 -- 172.21.15.37:0/2328515632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5412df40 con 0x7f4d540a2b30 2024-09-16T07:25:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d53fff700 1 -- 172.21.15.37:0/2328515632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d5409d580 con 0x7f4d540a2b30 2024-09-16T07:25:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 1 -- 172.21.15.37:0/2328515632 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f4d3c00cbb0 con 0x7f4d540a2b30 2024-09-16T07:25:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4d34035650 0x7f4d34037b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d4bfff700 1 -- 172.21.15.37:0/2328515632 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f4d3c047a00 con 0x7f4d540a2b30 2024-09-16T07:25:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d52ffd700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4d34035650 0x7f4d34037b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d52ffd700 10 monclient: get_auth_request con 0x7f4d34035650 auth_method 0 2024-09-16T07:25:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.938+0000 7f4d52ffd700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4d34035650 0x7f4d34037b10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009c60 tx=0x7f4d44008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.946+0000 7f4d4bfff700 1 -- 172.21.15.37:0/2328515632 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4d3c01bd80 con 0x7f4d540a2b30 2024-09-16T07:25:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.946+0000 7f4d4bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:00.946+0000 7f4d4bfff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:01.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.178+0000 7f4d53fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T07:25:01.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.178+0000 7f4d53fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T07:25:01.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.178+0000 7f4d53fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.178+0000 7f4d53fff700 1 -- 172.21.15.37:0/2328515632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4d540696e0 con 0x7f4d540a2b30 2024-09-16T07:25:01.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.178+0000 7f4d4bfff700 1 -- 172.21.15.37:0/2328515632 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1543 (secure 0 0 0) 0x7f4d3c017020 con 0x7f4d540a2b30 2024-09-16T07:25:01.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.178+0000 7f4d4bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T07:25:01.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.178+0000 7f4d4bfff700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:25:01.184 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","modified":"2024-09-16T07:18:06.498847Z","created":"2024-09-16T07:18:06.498847Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 -- 172.21.15.37:0/2328515632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4d34035650 msgr2=0x7f4d34037b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4d34035650 0x7f4d34037b10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009c60 tx=0x7f4d44008030 comp rx=0 tx=0).stop 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 10 monclient: shutdown 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 -- 172.21.15.37:0/2328515632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d540a2b30 msgr2=0x7f4d5412d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d540a2b30 0x7f4d5412d4c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00e9d0 tx=0x7f4d3c00ece0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 -- 172.21.15.37:0/2328515632 shutdown_connections 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4d34035650 0x7f4d34037b10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d54132010 0x7f4d5412da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d540a2b30 0x7f4d5412d4c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 --2- 172.21.15.37:0/2328515632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d540a2750 0x7f4d5413a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 -- 172.21.15.37:0/2328515632 >> 172.21.15.37:0/2328515632 conn(0x7f4d540a3ea0 msgr2=0x7f4d5409fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:01.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 -- 172.21.15.37:0/2328515632 shutdown_connections 2024-09-16T07:25:01.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.182+0000 7f4d53fff700 1 -- 172.21.15.37:0/2328515632 wait complete. 2024-09-16T07:25:01.197 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T07:25:01.197 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T07:25:01.208 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T07:25:01.208 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:25:01.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient: get_monmap_and_config 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient: build_initial_monmap 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient: monmap: 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:01.322178+0000 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:01.322178+0000 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:01.323 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:01.324 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:01.324 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 1 Processor -- start 2024-09-16T07:25:01.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 1 -- start start 2024-09-16T07:25:01.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient: init 2024-09-16T07:25:01.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:01.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T07:25:01.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee40a2540 0x7f2ee40a2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient(hunting): picked mon.noname-b con 0x7f2ee40a2540 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2ee40a3a60 0x7f2ee40a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient(hunting): picked mon.noname-c con 0x7f2ee40a3a60 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ee4135300 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40a1570 con 0x7f2ee4135300 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee40a33a0 con 0x7f2ee40a3a60 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee409a8b0 con 0x7f2ee40a2540 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.318+0000 7f2ee359e700 10 monclient(hunting): authenticate will time out at 1221.807983s 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee40a2540 0x7f2ee40a2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee2d9d700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2ee4135300 0x7f2ee4135710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54402/0 (socket says 172.21.15.37:54402) 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee2d9d700 1 -- 172.21.15.37:0/4095689268 learned_addr learned my addr 172.21.15.37:0/4095689268 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 10 monclient(hunting): get_auth_request con 0x7f2ee40a2540 auth_method 0 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee1d9b700 1 --2- 172.21.15.37:0/4095689268 >> v2:172.21.15.143:3300/0 conn(0x7f2ee40a3a60 0x7f2ee40a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee1d9b700 10 monclient(hunting): get_auth_request con 0x7f2ee40a3a60 auth_method 0 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 1 -- 172.21.15.37:0/4095689268 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795400782 0 0) 0x7f2ee40a1570 con 0x7f2ee4135300 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient(hunting): _negotiate 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient(hunting): my global_id is 54135 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 1 -- 172.21.15.37:0/4095689268 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed00035d0 con 0x7f2ee4135300 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 10 monclient(hunting): handle_auth_done global_id 44534 payload 274 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 1 -- 172.21.15.37:0/4095689268 >> v2:172.21.15.143:3300/0 conn(0x7f2ee40a3a60 msgr2=0x7f2ee40a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 1 --2- 172.21.15.37:0/4095689268 >> v2:172.21.15.143:3300/0 conn(0x7f2ee40a3a60 0x7f2ee40a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 1 -- 172.21.15.37:0/4095689268 >> v1:172.21.15.37:6789/0 conn(0x7f2ee4135300 legacy=0x7f2ee4135710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 10 monclient: _finish_hunting 0 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 1 monclient: found mon.noname-b 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 1 -- 172.21.15.37:0/4095689268 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee4099bf0 con 0x7f2ee40a2540 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee259c700 1 --2- 172.21.15.37:0/4095689268 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee40a2540 0x7f2ee40a2920 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009850 tx=0x7f2ecc009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1160c8ed8ccbf0a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 1 -- 172.21.15.37:0/4095689268 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc011830 con 0x7f2ee40a2540 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient: dump: 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.331 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.332 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.332 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.332 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.332 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.332 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.332 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.332 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.332 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient: _finish_auth 0 2024-09-16T07:25:01.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient: _check_auth_tickets 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 5 monclient: authenticate success, global_id 44534 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 1 -- 172.21.15.37:0/4095689268 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc011990 con 0x7f2ee40a2540 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 1 -- 172.21.15.37:0/4095689268 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc011d00 con 0x7f2ee40a2540 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 4 set_mon_vals no callback set 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee159a700 10 monclient: dump: 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.333 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 10 monclient: shutdown 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 1 -- 172.21.15.37:0/4095689268 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee40a2540 msgr2=0x7f2ee40a2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 1 --2- 172.21.15.37:0/4095689268 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee40a2540 0x7f2ee40a2920 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009850 tx=0x7f2ecc009b60 comp rx=0 tx=0).stop 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 1 -- 172.21.15.37:0/4095689268 shutdown_connections 2024-09-16T07:25:01.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 1 --2- 172.21.15.37:0/4095689268 >> v2:172.21.15.143:3300/0 conn(0x7f2ee40a3a60 0x7f2ee40a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 1 --2- 172.21.15.37:0/4095689268 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee40a2540 0x7f2ee40a2920 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 1 -- 172.21.15.37:0/4095689268 >> 172.21.15.37:0/4095689268 conn(0x7f2ee40a4640 msgr2=0x7f2ee40a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 1 -- 172.21.15.37:0/4095689268 shutdown_connections 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 1 -- 172.21.15.37:0/4095689268 wait complete. 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 10 monclient: build_initial_monmap 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.322+0000 7f2ee359e700 10 monclient: monmap: 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:01.335 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:01.336 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:01.336 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 Processor -- start 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 -- start start 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient: init 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee40a2540 0x7f2ee409faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient(hunting): picked mon.noname-b con 0x7f2ee40a2540 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ee40a3a60 0x7f2ee4133b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient(hunting): picked mon.noname-a con 0x7f2ee40a3a60 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee4135300 0x7f2ee41c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient(hunting): picked mon.noname-c con 0x7f2ee4135300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee4121560 con 0x7f2ee40a3a60 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee40a6ab0 con 0x7f2ee40a2540 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee406d650 con 0x7f2ee4135300 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient(hunting): authenticate will time out at 1221.813161s 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee4135300 0x7f2ee41c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ee40a3a60 0x7f2ee4133b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee4135300 0x7f2ee41c6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34920/0 (socket says 172.21.15.37:34920) 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ee40a3a60 0x7f2ee4133b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39302/0 (socket says 172.21.15.37:39302) 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 1 -- 172.21.15.37:0/1921758490 learned_addr learned my addr 172.21.15.37:0/1921758490 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 10 monclient(hunting): get_auth_request con 0x7f2ee4135300 auth_method 0 2024-09-16T07:25:01.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 10 monclient(hunting): get_auth_request con 0x7f2ee40a3a60 auth_method 0 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee259c700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee40a2540 0x7f2ee409faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee259c700 10 monclient(hunting): get_auth_request con 0x7f2ee40a2540 auth_method 0 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 10 monclient(hunting): handle_auth_done global_id 54138 payload 824 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 1 -- 172.21.15.37:0/1921758490 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee4135300 msgr2=0x7f2ee41c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee4135300 0x7f2ee41c6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 1 -- 172.21.15.37:0/1921758490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee40a2540 msgr2=0x7f2ee409faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee40a2540 0x7f2ee409faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 10 monclient: _finish_hunting 0 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 1 monclient: found mon.noname-a 2024-09-16T07:25:01.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 1 -- 172.21.15.37:0/1921758490 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc0094b0 con 0x7f2ee40a3a60 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee2d9d700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee4135300 0x7f2ee41c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee1d9b700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ee40a3a60 0x7f2ee4133b10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ed400efc0 tx=0x7f2ed400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 1 -- 172.21.15.37:0/1921758490 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed400cd10 con 0x7f2ee40a3a60 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 10 monclient: dump: 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.344 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 10 monclient: _finish_auth 0 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 10 monclient: _check_auth_tickets 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 1 -- 172.21.15.37:0/1921758490 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed400f040 con 0x7f2ee40a3a60 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 5 monclient: authenticate success, global_id 54138 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient: _renew_subs 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 -- 172.21.15.37:0/1921758490 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee4120c30 con 0x7f2ee40a3a60 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 1 -- 172.21.15.37:0/1921758490 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed40146a0 con 0x7f2ee40a3a60 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 10 monclient: dump: 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.346 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee0d99700 4 set_mon_vals no callback set 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient: _renew_subs 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 -- 172.21.15.37:0/1921758490 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee411fef0 con 0x7f2ee40a3a60 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2edaffd700 1 -- 172.21.15.37:0/1921758490 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f2ed4014800 con 0x7f2ee40a3a60 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.326+0000 7f2ee359e700 1 -- 172.21.15.37:0/1921758490 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee40a2920 con 0x7f2ee40a3a60 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.330+0000 7f2edaffd700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2ec4035600 0x7f2ec4037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.330+0000 7f2edaffd700 1 -- 172.21.15.37:0/1921758490 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f2ed4048500 con 0x7f2ee40a3a60 2024-09-16T07:25:01.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.330+0000 7f2ee259c700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2ec4035600 0x7f2ec4037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.330+0000 7f2ee259c700 10 monclient: get_auth_request con 0x7f2ec4035600 auth_method 0 2024-09-16T07:25:01.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.330+0000 7f2ee259c700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2ec4035600 0x7f2ec4037ac0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc0090b0 tx=0x7f2ecc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:01.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.334+0000 7f2edaffd700 1 -- 172.21.15.37:0/1921758490 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2ed4007b90 con 0x7f2ee40a3a60 2024-09-16T07:25:01.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.334+0000 7f2edaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:01.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.334+0000 7f2edaffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:01.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.510+0000 7f2ee359e700 1 -- 172.21.15.37:0/1921758490 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ee40a1bb0 con 0x7f2ec4035600 2024-09-16T07:25:01.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.514+0000 7f2edaffd700 1 -- 172.21.15.37:0/1921758490 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53349 (secure 0 0 0) 0x7f2ee40a1bb0 con 0x7f2ec4035600 2024-09-16T07:25:01.519 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 -- 172.21.15.37:0/1921758490 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2ec4035600 msgr2=0x7f2ec4037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2ec4035600 0x7f2ec4037ac0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc0090b0 tx=0x7f2ecc008030 comp rx=0 tx=0).stop 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 10 monclient: shutdown 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 -- 172.21.15.37:0/1921758490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ee40a3a60 msgr2=0x7f2ee4133b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ee40a3a60 0x7f2ee4133b10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ed400efc0 tx=0x7f2ed400c3a0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 -- 172.21.15.37:0/1921758490 shutdown_connections 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2ec4035600 0x7f2ec4037ac0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ee4135300 0x7f2ee41c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ee40a3a60 0x7f2ee4133b10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 --2- 172.21.15.37:0/1921758490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee40a2540 0x7f2ee409faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 -- 172.21.15.37:0/1921758490 >> 172.21.15.37:0/1921758490 conn(0x7f2ee40a4640 msgr2=0x7f2ee40a5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:01.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 -- 172.21.15.37:0/1921758490 shutdown_connections 2024-09-16T07:25:01.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.518+0000 7f2ee359e700 1 -- 172.21.15.37:0/1921758490 wait complete. 2024-09-16T07:25:01.522 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T07:25:01.535 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-09-16T07:25:00.554207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":196056,"kb_used_data":25744,"kb_used_omap":9,"kb_used_meta":170230,"kb_avail":566034984,"statfs":{"total":579820584960,"available":579619823616,"internally_reserved":0,"allocated":26361856,"data_stored":10445776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":174316198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":282,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.759121+0000","last_change":"2024-09-16T07:18:57.472278+0000","last_active":"2024-09-16T07:24:39.759121+0000","last_peered":"2024-09-16T07:24:39.759121+0000","last_clean":"2024-09-16T07:24:39.759121+0000","last_became_active":"2024-09-16T07:18:55.463443+0000","last_became_peered":"2024-09-16T07:18:55.463443+0000","last_unstale":"2024-09-16T07:24:39.759121+0000","last_undegraded":"2024-09-16T07:24:39.759121+0000","last_fullsized":"2024-09-16T07:24:39.759121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:10.660959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059939100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":286,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.768556+0000","last_change":"2024-09-16T07:18:57.474462+0000","last_active":"2024-09-16T07:24:39.768556+0000","last_peered":"2024-09-16T07:24:39.768556+0000","last_clean":"2024-09-16T07:24:39.768556+0000","last_became_active":"2024-09-16T07:18:55.467864+0000","last_became_peered":"2024-09-16T07:18:55.467864+0000","last_unstale":"2024-09-16T07:24:39.768556+0000","last_undegraded":"2024-09-16T07:24:39.768556+0000","last_fullsized":"2024-09-16T07:24:39.768556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:02.070207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066039799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":282,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.759015+0000","last_change":"2024-09-16T07:18:57.472359+0000","last_active":"2024-09-16T07:24:39.759015+0000","last_peered":"2024-09-16T07:24:39.759015+0000","last_clean":"2024-09-16T07:24:39.759015+0000","last_became_active":"2024-09-16T07:18:55.471978+0000","last_became_peered":"2024-09-16T07:18:55.471978+0000","last_unstale":"2024-09-16T07:24:39.759015+0000","last_undegraded":"2024-09-16T07:24:39.759015+0000","last_fullsized":"2024-09-16T07:24:39.759015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:36.770184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077051799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":283,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.763092+0000","last_change":"2024-09-16T07:18:57.464442+0000","last_active":"2024-09-16T07:24:39.763092+0000","last_peered":"2024-09-16T07:24:39.763092+0000","last_clean":"2024-09-16T07:24:39.763092+0000","last_became_active":"2024-09-16T07:18:55.468584+0000","last_became_peered":"2024-09-16T07:18:55.468584+0000","last_unstale":"2024-09-16T07:24:39.763092+0000","last_undegraded":"2024-09-16T07:24:39.763092+0000","last_fullsized":"2024-09-16T07:24:39.763092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:20.380838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":291,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.768039+0000","last_change":"2024-09-16T07:18:57.470967+0000","last_active":"2024-09-16T07:24:39.768039+0000","last_peered":"2024-09-16T07:24:39.768039+0000","last_clean":"2024-09-16T07:24:39.768039+0000","last_became_active":"2024-09-16T07:18:55.472671+0000","last_became_peered":"2024-09-16T07:18:55.472671+0000","last_unstale":"2024-09-16T07:24:39.768039+0000","last_undegraded":"2024-09-16T07:24:39.768039+0000","last_fullsized":"2024-09-16T07:24:39.768039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:13.933764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000840762,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":298,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.767524+0000","last_change":"2024-09-16T07:18:57.740277+0000","last_active":"2024-09-16T07:24:39.767524+0000","last_peered":"2024-09-16T07:24:39.767524+0000","last_clean":"2024-09-16T07:24:39.767524+0000","last_became_active":"2024-09-16T07:18:55.469151+0000","last_became_peered":"2024-09-16T07:18:55.469151+0000","last_unstale":"2024-09-16T07:24:39.767524+0000","last_undegraded":"2024-09-16T07:24:39.767524+0000","last_fullsized":"2024-09-16T07:24:39.767524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:34.482147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064042899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":302,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:42.293038+0000","last_change":"2024-09-16T07:18:57.743569+0000","last_active":"2024-09-16T07:24:42.293038+0000","last_peered":"2024-09-16T07:24:42.293038+0000","last_clean":"2024-09-16T07:24:42.293038+0000","last_became_active":"2024-09-16T07:18:55.469565+0000","last_became_peered":"2024-09-16T07:18:55.469565+0000","last_unstale":"2024-09-16T07:24:42.293038+0000","last_undegraded":"2024-09-16T07:24:42.293038+0000","last_fullsized":"2024-09-16T07:24:42.293038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:46.360051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077454399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":283,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.758756+0000","last_change":"2024-09-16T07:18:57.472930+0000","last_active":"2024-09-16T07:24:39.758756+0000","last_peered":"2024-09-16T07:24:39.758756+0000","last_clean":"2024-09-16T07:24:39.758756+0000","last_became_active":"2024-09-16T07:18:55.464362+0000","last_became_peered":"2024-09-16T07:18:55.464362+0000","last_unstale":"2024-09-16T07:24:39.758756+0000","last_undegraded":"2024-09-16T07:24:39.758756+0000","last_fullsized":"2024-09-16T07:24:39.758756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:51.926284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212957,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":431,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:40.679278+0000","last_change":"2024-09-16T07:18:49.375792+0000","last_active":"2024-09-16T07:24:40.679278+0000","last_peered":"2024-09-16T07:24:40.679278+0000","last_clean":"2024-09-16T07:24:40.679278+0000","last_became_active":"2024-09-16T07:18:49.373551+0000","last_became_peered":"2024-09-16T07:18:49.373551+0000","last_unstale":"2024-09-16T07:24:40.679278+0000","last_undegraded":"2024-09-16T07:24:40.679278+0000","last_fullsized":"2024-09-16T07:24:40.679278+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_clean_scrub_stamp":"2024-09-16T07:18:47.386541+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:31.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35232,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94336608,"statfs":{"total":96636764160,"available":96600686592,"internally_reserved":0,"allocated":4214784,"data_stored":1564647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.80700000000000005,"15min":0.84299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.99299999999999999,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.879},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.80400000000000005,"15min":0.81299999999999994},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.95799999999999996,"5min":1.329,"15min":1.329},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.84899999999999998,"15min":0.82899999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.048,"5min":2.177,"15min":2.177},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.84699999999999998,"15min":0.879},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0089999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.81100000000000005}]},{"osd":2,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.82599999999999996},"min":{"1min":0.53300000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0820000000000001,"5min":1.23,"15min":1.23},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.93300000000000005,"15min":0.89500000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.9369999999999998,"5min":3.8610000000000002,"15min":3.8610000000000002},"last":0.95999999999999996}]},{"osd":4,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.71699999999999997,"15min":0.72099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94699999999999995,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.752,"5min":0.76200000000000001,"15min":0.78600000000000003},"min":{"1min":0.48999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.133,"5min":1.351,"15min":1.351},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.74299999999999999,"15min":0.751},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.2050000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.89400000000000002,"15min":0.86199999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.92000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31136,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340704,"statfs":{"total":96636764160,"available":96604880896,"internally_reserved":0,"allocated":4214784,"data_stored":1564647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57199999999999995,"15min":0.57299999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68300000000000005,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64900000000000002,"15min":0.629},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80800000000000005,"5min":1.135,"15min":1.135},"last":0.65000000000000002}]},{"osd":1,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63500000000000001,"15min":0.65100000000000002},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.95899999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.622},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.72999999999999998,"15min":0.69199999999999995},"min":{"1min":0.43099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.079,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.68899999999999995}]},{"osd":3,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79000000000000004,"15min":0.77300000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.012,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.89000000000000001,"15min":0.85999999999999999},"min":{"1min":0.56200000000000006,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.76900000000000002}]},{"osd":4,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.92200000000000004,"15min":0.91400000000000003},"min":{"1min":0.51400000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.29,"5min":1.331,"15min":1.331},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.86299999999999999,"15min":0.90000000000000002},"min":{"1min":0.52900000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0740000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.89600000000000002}]},{"osd":5,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.90800000000000003,"15min":0.91700000000000004},"min":{"1min":0.58599999999999997,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.91800000000000004,"15min":0.93899999999999995},"min":{"1min":0.60599999999999998,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.196,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.80500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35236,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94336604,"statfs":{"total":96636764160,"available":96600682496,"internally_reserved":0,"allocated":4218880,"data_stored":1564666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.41599999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88400000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65000000000000002,"15min":0.65500000000000003},"min":{"1min":0.42099999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91900000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68400000000000005}]},{"osd":2,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60399999999999998,"15min":0.63300000000000001},"min":{"1min":0.39000000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72299999999999998,"15min":0.75700000000000001},"min":{"1min":0.52600000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.92200000000000004,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.752}]},{"osd":3,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.85799999999999998,"15min":0.86399999999999999},"min":{"1min":0.63800000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.181,"5min":1.611,"15min":1.611},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.69099999999999995,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.137,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.86599999999999999}]},{"osd":4,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.874,"15min":0.91100000000000003},"min":{"1min":0.67500000000000004,"5min":0.621,"15min":0.621},"max":{"1min":1.472,"5min":1.998,"15min":1.998},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.77500000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.264,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.98699999999999999}]},{"osd":5,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.91300000000000003,"15min":0.98599999999999999},"min":{"1min":0.64000000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.21,"5min":1.673,"15min":1.673},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97499999999999998,"15min":0.99299999999999999},"min":{"1min":0.80100000000000005,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.173,"5min":1.825,"15min":1.825},"last":1.1399999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607629,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31140,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340700,"statfs":{"total":96636764160,"available":96604876800,"internally_reserved":0,"allocated":4218880,"data_stored":1564666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.81799999999999995,"15min":0.84399999999999997},"min":{"1min":0.60399999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.105,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.83699999999999997,"15min":0.82299999999999995},"min":{"1min":0.76500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.204,"5min":1.272,"15min":1.272},"last":0.90900000000000003}]},{"osd":1,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.79500000000000004,"15min":0.81699999999999995},"min":{"1min":0.57499999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0629999999999999,"5min":1.462,"15min":1.462},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83499999999999996,"15min":0.86299999999999999},"min":{"1min":0.61199999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.149,"5min":1.548,"15min":1.548},"last":0.97899999999999998}]},{"osd":2,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.81599999999999995,"15min":0.78000000000000003},"min":{"1min":0.61199999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.246,"5min":2.246,"15min":2.246},"last":0.878},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.879,"15min":0.89300000000000002},"min":{"1min":0.65700000000000003,"5min":0.435,"15min":0.435},"max":{"1min":1.1899999999999999,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.68600000000000005}]},{"osd":3,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.77700000000000002,"15min":0.82499999999999996},"min":{"1min":0.48999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.117,"5min":2.423,"15min":2.423},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.54500000000000004,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.74099999999999999}]},{"osd":5,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.82799999999999996},"min":{"1min":0.56699999999999995,"5min":0.375,"15min":0.375},"max":{"1min":0.99199999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.86299999999999999,"15min":0.88},"min":{"1min":0.64600000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.206,"5min":1.776,"15min":1.776},"last":0.79300000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607628,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":4747264,"data_stored":2093575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.45000000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.78400000000000003,"5min":1.107,"15min":1.107},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.63800000000000001,"15min":0.66500000000000004},"min":{"1min":0.45000000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84999999999999998,"5min":1.23,"15min":1.23},"last":0.746}]},{"osd":2,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.66400000000000003,"15min":0.64100000000000001},"min":{"1min":0.437,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82699999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.499,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.94399999999999995,"5min":2.133,"15min":2.133},"last":0.84199999999999997}]},{"osd":3,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84199999999999997,"15min":0.83899999999999997},"min":{"1min":0.621,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.002,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88900000000000001,"15min":0.91800000000000004},"min":{"1min":0.67400000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.155,"5min":1.371,"15min":1.371},"last":0.88200000000000001}]},{"osd":4,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.92500000000000004,"15min":0.93200000000000005},"min":{"1min":0.65300000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0620000000000001,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.044},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.88900000000000001,"15min":0.91300000000000003},"min":{"1min":0.60399999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.94299999999999995,"5min":1.484,"15min":1.484},"last":0.95399999999999996}]},{"osd":5,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.90800000000000003,"15min":0.92000000000000004},"min":{"1min":0.67800000000000005,"5min":0.441,"15min":0.441},"max":{"1min":1.1200000000000001,"5min":1.393,"15min":1.393},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.997,"15min":1.038},"min":{"1min":0.76200000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.234,"5min":2.028,"15min":2.028},"last":0.98299999999999998}]}]},{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":4747264,"data_stored":2093575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.74399999999999999,"15min":0.72299999999999998},"min":{"1min":0.57899999999999996,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98799999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72899999999999998,"15min":0.745},"min":{"1min":0.59199999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.026,"5min":1.036,"15min":1.036},"last":0.77000000000000002}]},{"osd":1,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.82399999999999995,"15min":0.80400000000000005},"min":{"1min":0.73199999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.97699999999999998,"5min":1.254,"15min":1.254},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.86599999999999999,"15min":0.84799999999999998},"min":{"1min":0.68300000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.1020000000000001,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.748}]},{"osd":2,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85499999999999998,"15min":0.82999999999999996},"min":{"1min":0.59099999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98799999999999999,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.879},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.65400000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96299999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.96299999999999997}]},{"osd":3,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.70299999999999996,"15min":0.70899999999999996},"min":{"1min":0.53000000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.85599999999999998,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77700000000000002,"15min":0.78800000000000003},"min":{"1min":0.44,"5min":0.376,"15min":0.376},"max":{"1min":1.131,"5min":1.165,"15min":1.165},"last":0.621}]},{"osd":4,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.82699999999999996,"15min":0.81000000000000005},"min":{"1min":0.497,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1759999999999999,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.82099999999999995},"min":{"1min":0.53900000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0700000000000001,"5min":2.569,"15min":2.569},"last":0.71199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:25:01.537 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:25:01.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient: get_monmap_and_config 2024-09-16T07:25:01.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient: build_initial_monmap 2024-09-16T07:25:01.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient: monmap: 2024-09-16T07:25:01.652 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:01.652 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.652 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:01.651579+0000 2024-09-16T07:25:01.652 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:01.651579+0000 2024-09-16T07:25:01.653 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:01.653 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.653 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:01.653 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:01.653 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:01.653 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:01.653 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:01.653 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 1 Processor -- start 2024-09-16T07:25:01.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 1 -- start start 2024-09-16T07:25:01.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient: init 2024-09-16T07:25:01.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:01.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T07:25:01.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46740a5bb0 addr v1:172.21.15.143:6789/0 2024-09-16T07:25:01.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46740a5780 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:01.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467413bc60 0x7f467413e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient(hunting): picked mon.noname-b con 0x7f467413bc60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:01.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467409a5a0 con 0x7f46740a5780 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674099d00 con 0x7f46740a5bb0 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46740a16f0 con 0x7f467413bc60 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 10 monclient(hunting): authenticate will time out at 1222.137395s 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4671d9b700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f46740a5780 0x7f467413a540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54416/0 (socket says 172.21.15.37:54416) 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4671d9b700 1 -- 172.21.15.37:0/504236758 learned_addr learned my addr 172.21.15.37:0/504236758 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467413bc60 0x7f467413e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): get_auth_request con 0x7f467413bc60 auth_method 0 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507246863 0 0) 0x7f467409a5a0 con 0x7f46740a5780 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): _negotiate 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): my global_id is 54141 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f465c0035d0 con 0x7f46740a5780 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 723802491 0 0) 0x7f465c0035d0 con 0x7f46740a5780 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467413bc60 msgr2=0x7f467413e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:01.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467413bc60 0x7f467413e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 >> v1:172.21.15.143:6789/0 conn(0x7f46740a5bb0 legacy=0x7f46740a5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient: _finish_hunting 0 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 monclient: found mon.noname-a-legacy 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4674099fe0 con 0x7f46740a5780 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient: _finish_auth 0 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient: _check_auth_tickets 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 120013200 0 0) 0x7f46640019f0 con 0x7f46740a5780 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 5 monclient: authenticate success, global_id 54141 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:01.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient: dump: 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-09-16T07:25:01.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 >> v1:172.21.15.37:6789/0 conn(0x7f46740a5780 legacy=0x7f467413a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f465c003a50 0x7f465c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): picked mon.a con 0x7f465c003a50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467413bc60 0x7f465c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): picked mon.c con 0x7f467413bc60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c004db0 0x7f465c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): picked mon.b con 0x7f465c004db0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46740a16f0 con 0x7f465c003a50 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c00bc20 con 0x7f465c004db0 2024-09-16T07:25:01.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465c00bde0 con 0x7f467413bc60 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467159a700 10 monclient(hunting): _finish_auth 0 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467359e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c004db0 0x7f465c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): get_auth_request con 0x7f465c004db0 auth_method 0 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f465c003a50 0x7f465c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4671d9b700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467413bc60 0x7f465c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4671d9b700 10 monclient(hunting): get_auth_request con 0x7f467413bc60 auth_method 0 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4671d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4671d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4671d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient(hunting): get_auth_request con 0x7f465c003a50 auth_method 0 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4671d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4671d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4671d9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient(hunting): handle_auth_done global_id 54141 payload 293 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 1 -- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467413bc60 msgr2=0x7f465c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467413bc60 0x7f465c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 1 -- 172.21.15.37:0/504236758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c004db0 msgr2=0x7f465c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c004db0 0x7f465c005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient: _finish_hunting 0 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 1 monclient: found mon.a 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 1 -- 172.21.15.37:0/504236758 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c0035d0 con 0x7f465c003a50 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient: _finish_auth 0 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 10 monclient: _check_auth_tickets 2024-09-16T07:25:01.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f4672d9d700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c004db0 0x7f465c005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.650+0000 7f467259c700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f465c003a50 0x7f465c003e70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f46680017e0 tx=0x7f465c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f1ec5a19e1199f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46680052d0 con 0x7f465c003a50 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467159a700 10 monclient: dump: 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.675 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4668005430 con 0x7f465c003a50 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467159a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 4 set_mon_vals no callback set 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467159a700 1 -- 172.21.15.37:0/504236758 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46680055a0 con 0x7f465c003a50 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467159a700 10 monclient: dump: 2024-09-16T07:25:01.676 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient: shutdown 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:01.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 -- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f465c003a50 msgr2=0x7f465c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 --2- 172.21.15.37:0/504236758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f465c003a50 0x7f465c003e70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f46680017e0 tx=0x7f465c0058a0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 -- 172.21.15.37:0/504236758 reap_dead start 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 -- 172.21.15.37:0/504236758 shutdown_connections 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 -- 172.21.15.37:0/504236758 >> 172.21.15.37:0/504236758 conn(0x7f46740a4480 msgr2=0x7f46740a4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 -- 172.21.15.37:0/504236758 shutdown_connections 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 -- 172.21.15.37:0/504236758 wait complete. 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient: build_initial_monmap 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient: monmap: 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:01.678 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 Processor -- start 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 -- start start 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient: init 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c003a50 0x7f467411e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient(hunting): picked mon.noname-b con 0x7f465c003a50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467409ee90 0x7f46741342f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient(hunting): picked mon.noname-c con 0x7f467409ee90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4674134830 0x7f46741c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient(hunting): picked mon.noname-a con 0x7f4674134830 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467411f3f0 con 0x7f4674134830 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46740a3670 con 0x7f465c003a50 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467406d4a0 con 0x7f467409ee90 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467359e700 10 monclient(hunting): authenticate will time out at 1222.142550s 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4674134830 0x7f46741c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4671d9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467409ee90 0x7f46741342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4674134830 0x7f46741c9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39322/0 (socket says 172.21.15.37:39322) 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 -- 172.21.15.37:0/624704193 learned_addr learned my addr 172.21.15.37:0/624704193 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 10 monclient(hunting): get_auth_request con 0x7f4674134830 auth_method 0 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4671d9b700 10 monclient(hunting): get_auth_request con 0x7f467409ee90 auth_method 0 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4671d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4671d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4671d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467259c700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c003a50 0x7f467411e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4671d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4671d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4671d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467259c700 10 monclient(hunting): get_auth_request con 0x7f465c003a50 auth_method 0 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467259c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 10 monclient(hunting): handle_auth_done global_id 54144 payload 824 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 -- 172.21.15.37:0/624704193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467409ee90 msgr2=0x7f46741342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467409ee90 0x7f46741342f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 -- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c003a50 msgr2=0x7f467411e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c003a50 0x7f467411e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 10 monclient: _finish_hunting 0 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 monclient: found mon.noname-a 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 -- 172.21.15.37:0/624704193 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46740a3b10 con 0x7f4674134830 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f467259c700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c003a50 0x7f467411e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4671d9b700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467409ee90 0x7f46741342f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.654+0000 7f4672d9d700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4674134830 0x7f46741c9f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f46741362e0 tx=0x7f466c011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 1 -- 172.21.15.37:0/624704193 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c004020 con 0x7f4674134830 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 10 monclient: dump: 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.683 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 10 monclient: _finish_auth 0 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 5 monclient: authenticate success, global_id 54144 2024-09-16T07:25:01.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 1 -- 172.21.15.37:0/624704193 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c004180 con 0x7f4674134830 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 10 monclient: _renew_subs 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 1 -- 172.21.15.37:0/624704193 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46740a04e0 con 0x7f4674134830 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 1 -- 172.21.15.37:0/624704193 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c004510 con 0x7f4674134830 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 10 monclient: _renew_subs 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 1 -- 172.21.15.37:0/624704193 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46741ca490 con 0x7f4674134830 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4670d99700 4 set_mon_vals no callback set 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 10 monclient: dump: 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.685 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467359e700 1 -- 172.21.15.37:0/624704193 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46740a5f90 con 0x7f4674134830 2024-09-16T07:25:01.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 1 -- 172.21.15.37:0/624704193 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f466c00f8d0 con 0x7f4674134830 2024-09-16T07:25:01.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4654035600 0x7f4654037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f4662ffd700 1 -- 172.21.15.37:0/624704193 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f466c047800 con 0x7f4674134830 2024-09-16T07:25:01.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467259c700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4654035600 0x7f4654037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467259c700 10 monclient: get_auth_request con 0x7f4654035600 auth_method 0 2024-09-16T07:25:01.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.658+0000 7f467259c700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4654035600 0x7f4654037ac0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4668000c00 tx=0x7f4668009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:01.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.662+0000 7f4662ffd700 1 -- 172.21.15.37:0/624704193 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f466c009740 con 0x7f4674134830 2024-09-16T07:25:01.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.662+0000 7f4662ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:01.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.662+0000 7f4662ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:01.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.830+0000 7f467359e700 1 -- 172.21.15.37:0/624704193 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46740a72f0 con 0x7f4654035600 2024-09-16T07:25:01.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.834+0000 7f4662ffd700 1 -- 172.21.15.37:0/624704193 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53349 (secure 0 0 0) 0x7f4668001290 con 0x7f4654035600 2024-09-16T07:25:01.839 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:25:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 -- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4654035600 msgr2=0x7f4654037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4654035600 0x7f4654037ac0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4668000c00 tx=0x7f4668009030 comp rx=0 tx=0).stop 2024-09-16T07:25:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 10 monclient: shutdown 2024-09-16T07:25:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 -- 172.21.15.37:0/624704193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4674134830 msgr2=0x7f46741c9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4674134830 0x7f46741c9f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f46741362e0 tx=0x7f466c011eb0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 -- 172.21.15.37:0/624704193 shutdown_connections 2024-09-16T07:25:01.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4674134830 0x7f46741c9f50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f4654035600 0x7f4654037ac0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f467409ee90 0x7f46741342f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 --2- 172.21.15.37:0/624704193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f465c003a50 0x7f467411e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 -- 172.21.15.37:0/624704193 >> 172.21.15.37:0/624704193 conn(0x7f46740a4480 msgr2=0x7f46740a2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:01.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 -- 172.21.15.37:0/624704193 shutdown_connections 2024-09-16T07:25:01.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.838+0000 7f467359e700 1 -- 172.21.15.37:0/624704193 wait complete. 2024-09-16T07:25:01.842 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T07:25:01.855 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-09-16T07:25:00.554207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":196056,"kb_used_data":25744,"kb_used_omap":9,"kb_used_meta":170230,"kb_avail":566034984,"statfs":{"total":579820584960,"available":579619823616,"internally_reserved":0,"allocated":26361856,"data_stored":10445776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":174316198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":282,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.759121+0000","last_change":"2024-09-16T07:18:57.472278+0000","last_active":"2024-09-16T07:24:39.759121+0000","last_peered":"2024-09-16T07:24:39.759121+0000","last_clean":"2024-09-16T07:24:39.759121+0000","last_became_active":"2024-09-16T07:18:55.463443+0000","last_became_peered":"2024-09-16T07:18:55.463443+0000","last_unstale":"2024-09-16T07:24:39.759121+0000","last_undegraded":"2024-09-16T07:24:39.759121+0000","last_fullsized":"2024-09-16T07:24:39.759121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:10.660959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059939100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":286,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.768556+0000","last_change":"2024-09-16T07:18:57.474462+0000","last_active":"2024-09-16T07:24:39.768556+0000","last_peered":"2024-09-16T07:24:39.768556+0000","last_clean":"2024-09-16T07:24:39.768556+0000","last_became_active":"2024-09-16T07:18:55.467864+0000","last_became_peered":"2024-09-16T07:18:55.467864+0000","last_unstale":"2024-09-16T07:24:39.768556+0000","last_undegraded":"2024-09-16T07:24:39.768556+0000","last_fullsized":"2024-09-16T07:24:39.768556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:02.070207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066039799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":282,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.759015+0000","last_change":"2024-09-16T07:18:57.472359+0000","last_active":"2024-09-16T07:24:39.759015+0000","last_peered":"2024-09-16T07:24:39.759015+0000","last_clean":"2024-09-16T07:24:39.759015+0000","last_became_active":"2024-09-16T07:18:55.471978+0000","last_became_peered":"2024-09-16T07:18:55.471978+0000","last_unstale":"2024-09-16T07:24:39.759015+0000","last_undegraded":"2024-09-16T07:24:39.759015+0000","last_fullsized":"2024-09-16T07:24:39.759015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:36.770184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077051799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":283,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.763092+0000","last_change":"2024-09-16T07:18:57.464442+0000","last_active":"2024-09-16T07:24:39.763092+0000","last_peered":"2024-09-16T07:24:39.763092+0000","last_clean":"2024-09-16T07:24:39.763092+0000","last_became_active":"2024-09-16T07:18:55.468584+0000","last_became_peered":"2024-09-16T07:18:55.468584+0000","last_unstale":"2024-09-16T07:24:39.763092+0000","last_undegraded":"2024-09-16T07:24:39.763092+0000","last_fullsized":"2024-09-16T07:24:39.763092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:20.380838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":291,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.768039+0000","last_change":"2024-09-16T07:18:57.470967+0000","last_active":"2024-09-16T07:24:39.768039+0000","last_peered":"2024-09-16T07:24:39.768039+0000","last_clean":"2024-09-16T07:24:39.768039+0000","last_became_active":"2024-09-16T07:18:55.472671+0000","last_became_peered":"2024-09-16T07:18:55.472671+0000","last_unstale":"2024-09-16T07:24:39.768039+0000","last_undegraded":"2024-09-16T07:24:39.768039+0000","last_fullsized":"2024-09-16T07:24:39.768039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:13.933764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000840762,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":298,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.767524+0000","last_change":"2024-09-16T07:18:57.740277+0000","last_active":"2024-09-16T07:24:39.767524+0000","last_peered":"2024-09-16T07:24:39.767524+0000","last_clean":"2024-09-16T07:24:39.767524+0000","last_became_active":"2024-09-16T07:18:55.469151+0000","last_became_peered":"2024-09-16T07:18:55.469151+0000","last_unstale":"2024-09-16T07:24:39.767524+0000","last_undegraded":"2024-09-16T07:24:39.767524+0000","last_fullsized":"2024-09-16T07:24:39.767524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:34.482147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064042899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":302,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:42.293038+0000","last_change":"2024-09-16T07:18:57.743569+0000","last_active":"2024-09-16T07:24:42.293038+0000","last_peered":"2024-09-16T07:24:42.293038+0000","last_clean":"2024-09-16T07:24:42.293038+0000","last_became_active":"2024-09-16T07:18:55.469565+0000","last_became_peered":"2024-09-16T07:18:55.469565+0000","last_unstale":"2024-09-16T07:24:42.293038+0000","last_undegraded":"2024-09-16T07:24:42.293038+0000","last_fullsized":"2024-09-16T07:24:42.293038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:46.360051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077454399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":283,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.758756+0000","last_change":"2024-09-16T07:18:57.472930+0000","last_active":"2024-09-16T07:24:39.758756+0000","last_peered":"2024-09-16T07:24:39.758756+0000","last_clean":"2024-09-16T07:24:39.758756+0000","last_became_active":"2024-09-16T07:18:55.464362+0000","last_became_peered":"2024-09-16T07:18:55.464362+0000","last_unstale":"2024-09-16T07:24:39.758756+0000","last_undegraded":"2024-09-16T07:24:39.758756+0000","last_fullsized":"2024-09-16T07:24:39.758756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:51.926284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212957,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":431,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:40.679278+0000","last_change":"2024-09-16T07:18:49.375792+0000","last_active":"2024-09-16T07:24:40.679278+0000","last_peered":"2024-09-16T07:24:40.679278+0000","last_clean":"2024-09-16T07:24:40.679278+0000","last_became_active":"2024-09-16T07:18:49.373551+0000","last_became_peered":"2024-09-16T07:18:49.373551+0000","last_unstale":"2024-09-16T07:24:40.679278+0000","last_undegraded":"2024-09-16T07:24:40.679278+0000","last_fullsized":"2024-09-16T07:24:40.679278+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_clean_scrub_stamp":"2024-09-16T07:18:47.386541+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:31.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35232,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94336608,"statfs":{"total":96636764160,"available":96600686592,"internally_reserved":0,"allocated":4214784,"data_stored":1564647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.80700000000000005,"15min":0.84299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.99299999999999999,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.879},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.80400000000000005,"15min":0.81299999999999994},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.95799999999999996,"5min":1.329,"15min":1.329},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.84899999999999998,"15min":0.82899999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.048,"5min":2.177,"15min":2.177},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.84699999999999998,"15min":0.879},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0089999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.81100000000000005}]},{"osd":2,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.82599999999999996},"min":{"1min":0.53300000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0820000000000001,"5min":1.23,"15min":1.23},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.93300000000000005,"15min":0.89500000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.9369999999999998,"5min":3.8610000000000002,"15min":3.8610000000000002},"last":0.95999999999999996}]},{"osd":4,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.71699999999999997,"15min":0.72099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94699999999999995,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.752,"5min":0.76200000000000001,"15min":0.78600000000000003},"min":{"1min":0.48999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.133,"5min":1.351,"15min":1.351},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.74299999999999999,"15min":0.751},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.2050000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.89400000000000002,"15min":0.86199999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.92000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31136,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340704,"statfs":{"total":96636764160,"available":96604880896,"internally_reserved":0,"allocated":4214784,"data_stored":1564647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57199999999999995,"15min":0.57299999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68300000000000005,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64900000000000002,"15min":0.629},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80800000000000005,"5min":1.135,"15min":1.135},"last":0.65000000000000002}]},{"osd":1,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63500000000000001,"15min":0.65100000000000002},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.95899999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.622},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.72999999999999998,"15min":0.69199999999999995},"min":{"1min":0.43099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.079,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.68899999999999995}]},{"osd":3,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79000000000000004,"15min":0.77300000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.012,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.89000000000000001,"15min":0.85999999999999999},"min":{"1min":0.56200000000000006,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.76900000000000002}]},{"osd":4,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.92200000000000004,"15min":0.91400000000000003},"min":{"1min":0.51400000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.29,"5min":1.331,"15min":1.331},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.86299999999999999,"15min":0.90000000000000002},"min":{"1min":0.52900000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0740000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.89600000000000002}]},{"osd":5,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.90800000000000003,"15min":0.91700000000000004},"min":{"1min":0.58599999999999997,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.91800000000000004,"15min":0.93899999999999995},"min":{"1min":0.60599999999999998,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.196,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.80500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35236,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94336604,"statfs":{"total":96636764160,"available":96600682496,"internally_reserved":0,"allocated":4218880,"data_stored":1564666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.41599999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88400000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65000000000000002,"15min":0.65500000000000003},"min":{"1min":0.42099999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91900000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68400000000000005}]},{"osd":2,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60399999999999998,"15min":0.63300000000000001},"min":{"1min":0.39000000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72299999999999998,"15min":0.75700000000000001},"min":{"1min":0.52600000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.92200000000000004,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.752}]},{"osd":3,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.85799999999999998,"15min":0.86399999999999999},"min":{"1min":0.63800000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.181,"5min":1.611,"15min":1.611},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.69099999999999995,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.137,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.86599999999999999}]},{"osd":4,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.874,"15min":0.91100000000000003},"min":{"1min":0.67500000000000004,"5min":0.621,"15min":0.621},"max":{"1min":1.472,"5min":1.998,"15min":1.998},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.77500000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.264,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.98699999999999999}]},{"osd":5,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.91300000000000003,"15min":0.98599999999999999},"min":{"1min":0.64000000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.21,"5min":1.673,"15min":1.673},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97499999999999998,"15min":0.99299999999999999},"min":{"1min":0.80100000000000005,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.173,"5min":1.825,"15min":1.825},"last":1.1399999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607629,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31140,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340700,"statfs":{"total":96636764160,"available":96604876800,"internally_reserved":0,"allocated":4218880,"data_stored":1564666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.81799999999999995,"15min":0.84399999999999997},"min":{"1min":0.60399999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.105,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.83699999999999997,"15min":0.82299999999999995},"min":{"1min":0.76500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.204,"5min":1.272,"15min":1.272},"last":0.90900000000000003}]},{"osd":1,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.79500000000000004,"15min":0.81699999999999995},"min":{"1min":0.57499999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0629999999999999,"5min":1.462,"15min":1.462},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83499999999999996,"15min":0.86299999999999999},"min":{"1min":0.61199999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.149,"5min":1.548,"15min":1.548},"last":0.97899999999999998}]},{"osd":2,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.81599999999999995,"15min":0.78000000000000003},"min":{"1min":0.61199999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.246,"5min":2.246,"15min":2.246},"last":0.878},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.879,"15min":0.89300000000000002},"min":{"1min":0.65700000000000003,"5min":0.435,"15min":0.435},"max":{"1min":1.1899999999999999,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.68600000000000005}]},{"osd":3,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.77700000000000002,"15min":0.82499999999999996},"min":{"1min":0.48999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.117,"5min":2.423,"15min":2.423},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.54500000000000004,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.74099999999999999}]},{"osd":5,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.82799999999999996},"min":{"1min":0.56699999999999995,"5min":0.375,"15min":0.375},"max":{"1min":0.99199999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.86299999999999999,"15min":0.88},"min":{"1min":0.64600000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.206,"5min":1.776,"15min":1.776},"last":0.79300000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607628,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":4747264,"data_stored":2093575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.45000000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.78400000000000003,"5min":1.107,"15min":1.107},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.63800000000000001,"15min":0.66500000000000004},"min":{"1min":0.45000000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84999999999999998,"5min":1.23,"15min":1.23},"last":0.746}]},{"osd":2,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.66400000000000003,"15min":0.64100000000000001},"min":{"1min":0.437,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82699999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.499,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.94399999999999995,"5min":2.133,"15min":2.133},"last":0.84199999999999997}]},{"osd":3,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84199999999999997,"15min":0.83899999999999997},"min":{"1min":0.621,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.002,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88900000000000001,"15min":0.91800000000000004},"min":{"1min":0.67400000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.155,"5min":1.371,"15min":1.371},"last":0.88200000000000001}]},{"osd":4,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.92500000000000004,"15min":0.93200000000000005},"min":{"1min":0.65300000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0620000000000001,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.044},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.88900000000000001,"15min":0.91300000000000003},"min":{"1min":0.60399999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.94299999999999995,"5min":1.484,"15min":1.484},"last":0.95399999999999996}]},{"osd":5,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.90800000000000003,"15min":0.92000000000000004},"min":{"1min":0.67800000000000005,"5min":0.441,"15min":0.441},"max":{"1min":1.1200000000000001,"5min":1.393,"15min":1.393},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.997,"15min":1.038},"min":{"1min":0.76200000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.234,"5min":2.028,"15min":2.028},"last":0.98299999999999998}]}]},{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":4747264,"data_stored":2093575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.74399999999999999,"15min":0.72299999999999998},"min":{"1min":0.57899999999999996,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98799999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72899999999999998,"15min":0.745},"min":{"1min":0.59199999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.026,"5min":1.036,"15min":1.036},"last":0.77000000000000002}]},{"osd":1,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.82399999999999995,"15min":0.80400000000000005},"min":{"1min":0.73199999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.97699999999999998,"5min":1.254,"15min":1.254},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.86599999999999999,"15min":0.84799999999999998},"min":{"1min":0.68300000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.1020000000000001,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.748}]},{"osd":2,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85499999999999998,"15min":0.82999999999999996},"min":{"1min":0.59099999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98799999999999999,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.879},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.65400000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96299999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.96299999999999997}]},{"osd":3,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.70299999999999996,"15min":0.70899999999999996},"min":{"1min":0.53000000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.85599999999999998,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77700000000000002,"15min":0.78800000000000003},"min":{"1min":0.44,"5min":0.376,"15min":0.376},"max":{"1min":1.131,"5min":1.165,"15min":1.165},"last":0.621}]},{"osd":4,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.82699999999999996,"15min":0.81000000000000005},"min":{"1min":0.497,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1759999999999999,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.82099999999999995},"min":{"1min":0.53900000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0700000000000001,"5min":2.569,"15min":2.569},"last":0.71199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:25:01.856 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T07:25:01.857 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:25:01.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient: get_monmap_and_config 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient: build_initial_monmap 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient: monmap: 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:01.964068+0000 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:01.964068+0000 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:01.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:01.966 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:01.966 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:01.966 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 1 Processor -- start 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 1 -- start start 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient: init 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbccc0a0e90 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbccc0a04d0 0x7fbccc0a1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient(hunting): picked mon.noname-a con 0x7fbccc0a04d0 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbccc0a2d50 0x7fbccc0a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient(hunting): picked mon.noname-c con 0x7fbccc0a2d50 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbccc0a0900 con 0x7fbccc0a0e90 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbccc0a17b0 con 0x7fbccc0a04d0 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbccc06cf50 con 0x7fbccc0a2d50 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient(hunting): authenticate will time out at 1222.450131s 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbccc0a04d0 0x7fbccc0a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbccad9d700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fbccc0a0e90 0x7fbccc0a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54420/0 (socket says 172.21.15.37:54420) 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbccad9d700 1 -- 172.21.15.37:0/1849576122 learned_addr learned my addr 172.21.15.37:0/1849576122 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 10 monclient(hunting): get_auth_request con 0x7fbccc0a04d0 auth_method 0 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbccb59e700 1 --2- 172.21.15.37:0/1849576122 >> v2:172.21.15.143:3300/0 conn(0x7fbccc0a2d50 0x7fbccc0a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbccb59e700 10 monclient(hunting): get_auth_request con 0x7fbccc0a2d50 auth_method 0 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbccb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbccb59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbccb59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbccb59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbccb59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbccb59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 10 monclient(hunting): handle_auth_done global_id 54147 payload 274 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 1 -- 172.21.15.37:0/1849576122 >> v2:172.21.15.143:3300/0 conn(0x7fbccc0a2d50 msgr2=0x7fbccc0a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 1 --2- 172.21.15.37:0/1849576122 >> v2:172.21.15.143:3300/0 conn(0x7fbccc0a2d50 0x7fbccc0a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 1 -- 172.21.15.37:0/1849576122 >> v1:172.21.15.37:6789/0 conn(0x7fbccc0a0e90 legacy=0x7fbccc0a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 10 monclient: _finish_hunting 0 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 1 monclient: found mon.noname-a 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 1 -- 172.21.15.37:0/1849576122 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc0a36c0 con 0x7fbccc0a04d0 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcca59c700 1 --2- 172.21.15.37:0/1849576122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbccc0a04d0 0x7fbccc0a1270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbccc09ff10 tx=0x7fbcbc009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1816a00a8a728d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 1 -- 172.21.15.37:0/1849576122 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc011830 con 0x7fbccc0a04d0 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 10 monclient: dump: 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.972 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 10 monclient: _finish_auth 0 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 10 monclient: _check_auth_tickets 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 5 monclient: authenticate success, global_id 54147 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 1 -- 172.21.15.37:0/1849576122 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcbc011990 con 0x7fbccc0a04d0 2024-09-16T07:25:01.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcd09c8700 4 set_mon_vals no callback set 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 1 -- 172.21.15.37:0/1849576122 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc011d20 con 0x7fbccc0a04d0 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.962+0000 7fbcc9d9b700 10 monclient: dump: 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.974 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient: shutdown 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 -- 172.21.15.37:0/1849576122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbccc0a04d0 msgr2=0x7fbccc0a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 --2- 172.21.15.37:0/1849576122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbccc0a04d0 0x7fbccc0a1270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbccc09ff10 tx=0x7fbcbc009b60 comp rx=0 tx=0).stop 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 -- 172.21.15.37:0/1849576122 shutdown_connections 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 --2- 172.21.15.37:0/1849576122 >> v2:172.21.15.143:3300/0 conn(0x7fbccc0a2d50 0x7fbccc0a3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 --2- 172.21.15.37:0/1849576122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbccc0a04d0 0x7fbccc0a1270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 -- 172.21.15.37:0/1849576122 >> 172.21.15.37:0/1849576122 conn(0x7fbccc0a4740 msgr2=0x7fbccc0a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 -- 172.21.15.37:0/1849576122 shutdown_connections 2024-09-16T07:25:01.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 -- 172.21.15.37:0/1849576122 wait complete. 2024-09-16T07:25:01.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient: build_initial_monmap 2024-09-16T07:25:01.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient: monmap: 2024-09-16T07:25:01.979 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:01.979 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 Processor -- start 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 -- start start 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient: init 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:01.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbccc0a04d0 0x7fbccc1c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient(hunting): picked mon.noname-c con 0x7fbccc0a04d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbccc0a0e90 0x7fbccc1c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient(hunting): picked mon.noname-b con 0x7fbccc0a0e90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbccc0a2d50 0x7fbccc1ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient(hunting): picked mon.noname-a con 0x7fbccc0a2d50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc09e250 con 0x7fbccc0a2d50 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc1c7c60 con 0x7fbccc0a0e90 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbccc06d780 con 0x7fbccc0a04d0 2024-09-16T07:25:01.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcd09c8700 10 monclient(hunting): authenticate will time out at 1222.455487s 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbccc0a04d0 0x7fbccc1c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbccc0a04d0 0x7fbccc1c6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34940/0 (socket says 172.21.15.37:34940) 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 1 -- 172.21.15.37:0/641695672 learned_addr learned my addr 172.21.15.37:0/641695672 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 10 monclient(hunting): get_auth_request con 0x7fbccc0a04d0 auth_method 0 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccb59e700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbccc0a2d50 0x7fbccc1ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccb59e700 10 monclient(hunting): get_auth_request con 0x7fbccc0a2d50 auth_method 0 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccb59e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccb59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcca59c700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbccc0a0e90 0x7fbccc1c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcca59c700 10 monclient(hunting): get_auth_request con 0x7fbccc0a0e90 auth_method 0 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcca59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcca59c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbcca59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccb59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccb59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccb59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.966+0000 7fbccad9d700 10 monclient(hunting): handle_auth_done global_id 44546 payload 824 2024-09-16T07:25:01.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbccad9d700 1 -- 172.21.15.37:0/641695672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbccc0a0e90 msgr2=0x7fbccc1c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbccad9d700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbccc0a0e90 0x7fbccc1c4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbccad9d700 1 -- 172.21.15.37:0/641695672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbccc0a2d50 msgr2=0x7fbccc1ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbccad9d700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbccc0a2d50 0x7fbccc1ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbccad9d700 10 monclient: _finish_hunting 0 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbccad9d700 1 monclient: found mon.noname-c 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbccad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbccad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbccad9d700 1 -- 172.21.15.37:0/641695672 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc0094b0 con 0x7fbccc0a04d0 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbccad9d700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbccc0a04d0 0x7fbccc1c6cf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000c730 tx=0x7fbcc000ca40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 1 -- 172.21.15.37:0/641695672 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0005d90 con 0x7fbccc0a04d0 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:01.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 10 monclient: dump: 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 10 monclient: _finish_auth 0 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 10 monclient: _check_auth_tickets 2024-09-16T07:25:01.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 1 -- 172.21.15.37:0/641695672 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc00070d0 con 0x7fbccc0a04d0 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 5 monclient: authenticate success, global_id 44546 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 10 monclient: _renew_subs 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 1 -- 172.21.15.37:0/641695672 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbccc1a90b0 con 0x7fbccc0a04d0 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 1 -- 172.21.15.37:0/641695672 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0007440 con 0x7fbccc0a04d0 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 10 monclient: dump: 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.986 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcc959a700 4 set_mon_vals no callback set 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 10 monclient: _renew_subs 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 1 -- 172.21.15.37:0/641695672 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbccc1ab340 con 0x7fbccc0a04d0 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:01.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:01.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcd09c8700 1 -- 172.21.15.37:0/641695672 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbccc0a1270 con 0x7fbccc0a04d0 2024-09-16T07:25:01.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 1 -- 172.21.15.37:0/641695672 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fbcc00075a0 con 0x7fbccc0a04d0 2024-09-16T07:25:01.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbcb8035600 0x7fbcb8037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:01.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcb37fe700 1 -- 172.21.15.37:0/641695672 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fbcc0046df0 con 0x7fbccc0a04d0 2024-09-16T07:25:01.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcca59c700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbcb8035600 0x7fbcb8037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:01.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcca59c700 10 monclient: get_auth_request con 0x7fbcb8035600 auth_method 0 2024-09-16T07:25:01.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.970+0000 7fbcca59c700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbcb8035600 0x7fbcb8037ac0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc009ba0 tx=0x7fbcbc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:01.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.978+0000 7fbcb37fe700 1 -- 172.21.15.37:0/641695672 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbcc0011d70 con 0x7fbccc0a04d0 2024-09-16T07:25:01.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.978+0000 7fbcb37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:01.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:01.978+0000 7fbcb37fe700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:02.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.146+0000 7fbcd09c8700 1 -- 172.21.15.37:0/641695672 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbccc06cbd0 con 0x7fbcb8035600 2024-09-16T07:25:02.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcb37fe700 1 -- 172.21.15.37:0/641695672 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53349 (secure 0 0 0) 0x7fbcbc011750 con 0x7fbcb8035600 2024-09-16T07:25:02.157 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:25:02.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 -- 172.21.15.37:0/641695672 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbcb8035600 msgr2=0x7fbcb8037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:02.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbcb8035600 0x7fbcb8037ac0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc009ba0 tx=0x7fbcbc008030 comp rx=0 tx=0).stop 2024-09-16T07:25:02.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 10 monclient: shutdown 2024-09-16T07:25:02.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:02.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 -- 172.21.15.37:0/641695672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbccc0a04d0 msgr2=0x7fbccc1c6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:02.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbccc0a04d0 0x7fbccc1c6cf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000c730 tx=0x7fbcc000ca40 comp rx=0 tx=0).stop 2024-09-16T07:25:02.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 -- 172.21.15.37:0/641695672 shutdown_connections 2024-09-16T07:25:02.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbccc0a2d50 0x7fbccc1ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbccc0a0e90 0x7fbccc1c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fbcb8035600 0x7fbcb8037ac0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 --2- 172.21.15.37:0/641695672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbccc0a04d0 0x7fbccc1c6cf0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 -- 172.21.15.37:0/641695672 >> 172.21.15.37:0/641695672 conn(0x7fbccc0a4740 msgr2=0x7fbccc0a71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:02.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 -- 172.21.15.37:0/641695672 shutdown_connections 2024-09-16T07:25:02.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.154+0000 7fbcd09c8700 1 -- 172.21.15.37:0/641695672 wait complete. 2024-09-16T07:25:02.160 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T07:25:02.172 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-09-16T07:25:00.554207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":196056,"kb_used_data":25744,"kb_used_omap":9,"kb_used_meta":170230,"kb_avail":566034984,"statfs":{"total":579820584960,"available":579619823616,"internally_reserved":0,"allocated":26361856,"data_stored":10445776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":174316198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":282,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.759121+0000","last_change":"2024-09-16T07:18:57.472278+0000","last_active":"2024-09-16T07:24:39.759121+0000","last_peered":"2024-09-16T07:24:39.759121+0000","last_clean":"2024-09-16T07:24:39.759121+0000","last_became_active":"2024-09-16T07:18:55.463443+0000","last_became_peered":"2024-09-16T07:18:55.463443+0000","last_unstale":"2024-09-16T07:24:39.759121+0000","last_undegraded":"2024-09-16T07:24:39.759121+0000","last_fullsized":"2024-09-16T07:24:39.759121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:10.660959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059939100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":286,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.768556+0000","last_change":"2024-09-16T07:18:57.474462+0000","last_active":"2024-09-16T07:24:39.768556+0000","last_peered":"2024-09-16T07:24:39.768556+0000","last_clean":"2024-09-16T07:24:39.768556+0000","last_became_active":"2024-09-16T07:18:55.467864+0000","last_became_peered":"2024-09-16T07:18:55.467864+0000","last_unstale":"2024-09-16T07:24:39.768556+0000","last_undegraded":"2024-09-16T07:24:39.768556+0000","last_fullsized":"2024-09-16T07:24:39.768556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:02.070207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066039799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":282,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.759015+0000","last_change":"2024-09-16T07:18:57.472359+0000","last_active":"2024-09-16T07:24:39.759015+0000","last_peered":"2024-09-16T07:24:39.759015+0000","last_clean":"2024-09-16T07:24:39.759015+0000","last_became_active":"2024-09-16T07:18:55.471978+0000","last_became_peered":"2024-09-16T07:18:55.471978+0000","last_unstale":"2024-09-16T07:24:39.759015+0000","last_undegraded":"2024-09-16T07:24:39.759015+0000","last_fullsized":"2024-09-16T07:24:39.759015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:36.770184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077051799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":283,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.763092+0000","last_change":"2024-09-16T07:18:57.464442+0000","last_active":"2024-09-16T07:24:39.763092+0000","last_peered":"2024-09-16T07:24:39.763092+0000","last_clean":"2024-09-16T07:24:39.763092+0000","last_became_active":"2024-09-16T07:18:55.468584+0000","last_became_peered":"2024-09-16T07:18:55.468584+0000","last_unstale":"2024-09-16T07:24:39.763092+0000","last_undegraded":"2024-09-16T07:24:39.763092+0000","last_fullsized":"2024-09-16T07:24:39.763092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:20.380838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":291,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.768039+0000","last_change":"2024-09-16T07:18:57.470967+0000","last_active":"2024-09-16T07:24:39.768039+0000","last_peered":"2024-09-16T07:24:39.768039+0000","last_clean":"2024-09-16T07:24:39.768039+0000","last_became_active":"2024-09-16T07:18:55.472671+0000","last_became_peered":"2024-09-16T07:18:55.472671+0000","last_unstale":"2024-09-16T07:24:39.768039+0000","last_undegraded":"2024-09-16T07:24:39.768039+0000","last_fullsized":"2024-09-16T07:24:39.768039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:13.933764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000840762,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":298,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.767524+0000","last_change":"2024-09-16T07:18:57.740277+0000","last_active":"2024-09-16T07:24:39.767524+0000","last_peered":"2024-09-16T07:24:39.767524+0000","last_clean":"2024-09-16T07:24:39.767524+0000","last_became_active":"2024-09-16T07:18:55.469151+0000","last_became_peered":"2024-09-16T07:18:55.469151+0000","last_unstale":"2024-09-16T07:24:39.767524+0000","last_undegraded":"2024-09-16T07:24:39.767524+0000","last_fullsized":"2024-09-16T07:24:39.767524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:34.482147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064042899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":302,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:42.293038+0000","last_change":"2024-09-16T07:18:57.743569+0000","last_active":"2024-09-16T07:24:42.293038+0000","last_peered":"2024-09-16T07:24:42.293038+0000","last_clean":"2024-09-16T07:24:42.293038+0000","last_became_active":"2024-09-16T07:18:55.469565+0000","last_became_peered":"2024-09-16T07:18:55.469565+0000","last_unstale":"2024-09-16T07:24:42.293038+0000","last_undegraded":"2024-09-16T07:24:42.293038+0000","last_fullsized":"2024-09-16T07:24:42.293038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:46.360051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077454399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":283,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.758756+0000","last_change":"2024-09-16T07:18:57.472930+0000","last_active":"2024-09-16T07:24:39.758756+0000","last_peered":"2024-09-16T07:24:39.758756+0000","last_clean":"2024-09-16T07:24:39.758756+0000","last_became_active":"2024-09-16T07:18:55.464362+0000","last_became_peered":"2024-09-16T07:18:55.464362+0000","last_unstale":"2024-09-16T07:24:39.758756+0000","last_undegraded":"2024-09-16T07:24:39.758756+0000","last_fullsized":"2024-09-16T07:24:39.758756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:51.926284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212957,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":431,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:40.679278+0000","last_change":"2024-09-16T07:18:49.375792+0000","last_active":"2024-09-16T07:24:40.679278+0000","last_peered":"2024-09-16T07:24:40.679278+0000","last_clean":"2024-09-16T07:24:40.679278+0000","last_became_active":"2024-09-16T07:18:49.373551+0000","last_became_peered":"2024-09-16T07:18:49.373551+0000","last_unstale":"2024-09-16T07:24:40.679278+0000","last_undegraded":"2024-09-16T07:24:40.679278+0000","last_fullsized":"2024-09-16T07:24:40.679278+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_clean_scrub_stamp":"2024-09-16T07:18:47.386541+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:42:31.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574926,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35232,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94336608,"statfs":{"total":96636764160,"available":96600686592,"internally_reserved":0,"allocated":4214784,"data_stored":1564647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.80700000000000005,"15min":0.84299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.99299999999999999,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.879},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.80400000000000005,"15min":0.81299999999999994},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.95799999999999996,"5min":1.329,"15min":1.329},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.84899999999999998,"15min":0.82899999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.048,"5min":2.177,"15min":2.177},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.84699999999999998,"15min":0.879},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.0089999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.81100000000000005}]},{"osd":2,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.83099999999999996,"15min":0.82599999999999996},"min":{"1min":0.53300000000000003,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0820000000000001,"5min":1.23,"15min":1.23},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.93300000000000005,"15min":0.89500000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.9369999999999998,"5min":3.8610000000000002,"15min":3.8610000000000002},"last":0.95999999999999996}]},{"osd":4,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.71699999999999997,"15min":0.72099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94699999999999995,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.752,"5min":0.76200000000000001,"15min":0.78600000000000003},"min":{"1min":0.48999999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.133,"5min":1.351,"15min":1.351},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Sep 16 07:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.74299999999999999,"15min":0.751},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.2050000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.89400000000000002,"15min":0.86199999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.92000000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31136,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340704,"statfs":{"total":96636764160,"available":96604880896,"internally_reserved":0,"allocated":4214784,"data_stored":1564647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57199999999999995,"15min":0.57299999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68300000000000005,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64900000000000002,"15min":0.629},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80800000000000005,"5min":1.135,"15min":1.135},"last":0.65000000000000002}]},{"osd":1,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63500000000000001,"15min":0.65100000000000002},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.95899999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.622},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.72999999999999998,"15min":0.69199999999999995},"min":{"1min":0.43099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.079,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.68899999999999995}]},{"osd":3,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79000000000000004,"15min":0.77300000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.012,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.89000000000000001,"15min":0.85999999999999999},"min":{"1min":0.56200000000000006,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.76900000000000002}]},{"osd":4,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.92200000000000004,"15min":0.91400000000000003},"min":{"1min":0.51400000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.29,"5min":1.331,"15min":1.331},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.86299999999999999,"15min":0.90000000000000002},"min":{"1min":0.52900000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0740000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.89600000000000002}]},{"osd":5,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.90800000000000003,"15min":0.91700000000000004},"min":{"1min":0.58599999999999997,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.91800000000000004,"15min":0.93899999999999995},"min":{"1min":0.60599999999999998,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.196,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.80500000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35236,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94336604,"statfs":{"total":96636764160,"available":96600682496,"internally_reserved":0,"allocated":4218880,"data_stored":1564666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.41599999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88400000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65000000000000002,"15min":0.65500000000000003},"min":{"1min":0.42099999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91900000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68400000000000005}]},{"osd":2,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.60399999999999998,"15min":0.63300000000000001},"min":{"1min":0.39000000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72299999999999998,"15min":0.75700000000000001},"min":{"1min":0.52600000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.92200000000000004,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.752}]},{"osd":3,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.85799999999999998,"15min":0.86399999999999999},"min":{"1min":0.63800000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.181,"5min":1.611,"15min":1.611},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.69099999999999995,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.137,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.86599999999999999}]},{"osd":4,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.874,"15min":0.91100000000000003},"min":{"1min":0.67500000000000004,"5min":0.621,"15min":0.621},"max":{"1min":1.472,"5min":1.998,"15min":1.998},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.77500000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.264,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.98699999999999999}]},{"osd":5,"last update":"Mon Sep 16 07:23:59 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.91300000000000003,"15min":0.98599999999999999},"min":{"1min":0.64000000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.21,"5min":1.673,"15min":1.673},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97499999999999998,"15min":0.99299999999999999},"min":{"1min":0.80100000000000005,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.173,"5min":1.825,"15min":1.825},"last":1.1399999999999999}]}]},{"osd":4,"up_from":12,"seq":51539607629,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31140,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340700,"statfs":{"total":96636764160,"available":96604876800,"internally_reserved":0,"allocated":4218880,"data_stored":1564666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.81799999999999995,"15min":0.84399999999999997},"min":{"1min":0.60399999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.105,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.83699999999999997,"15min":0.82299999999999995},"min":{"1min":0.76500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.204,"5min":1.272,"15min":1.272},"last":0.90900000000000003}]},{"osd":1,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.79500000000000004,"15min":0.81699999999999995},"min":{"1min":0.57499999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0629999999999999,"5min":1.462,"15min":1.462},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83499999999999996,"15min":0.86299999999999999},"min":{"1min":0.61199999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.149,"5min":1.548,"15min":1.548},"last":0.97899999999999998}]},{"osd":2,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.81599999999999995,"15min":0.78000000000000003},"min":{"1min":0.61199999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.246,"5min":2.246,"15min":2.246},"last":0.878},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.879,"15min":0.89300000000000002},"min":{"1min":0.65700000000000003,"5min":0.435,"15min":0.435},"max":{"1min":1.1899999999999999,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.68600000000000005}]},{"osd":3,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.77700000000000002,"15min":0.82499999999999996},"min":{"1min":0.48999999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.117,"5min":2.423,"15min":2.423},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78800000000000003,"15min":0.79300000000000004},"min":{"1min":0.54500000000000004,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.74099999999999999}]},{"osd":5,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.80900000000000005,"15min":0.82799999999999996},"min":{"1min":0.56699999999999995,"5min":0.375,"15min":0.375},"max":{"1min":0.99199999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.86299999999999999,"15min":0.88},"min":{"1min":0.64600000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.206,"5min":1.776,"15min":1.776},"last":0.79300000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607628,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":4747264,"data_stored":2093575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.45000000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.78400000000000003,"5min":1.107,"15min":1.107},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.63800000000000001,"15min":0.66500000000000004},"min":{"1min":0.45000000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84999999999999998,"5min":1.23,"15min":1.23},"last":0.746}]},{"osd":2,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.66400000000000003,"15min":0.64100000000000001},"min":{"1min":0.437,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82699999999999996,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69999999999999996,"15min":0.73999999999999999},"min":{"1min":0.499,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.94399999999999995,"5min":2.133,"15min":2.133},"last":0.84199999999999997}]},{"osd":3,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84199999999999997,"15min":0.83899999999999997},"min":{"1min":0.621,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.002,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88900000000000001,"15min":0.91800000000000004},"min":{"1min":0.67400000000000004,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.155,"5min":1.371,"15min":1.371},"last":0.88200000000000001}]},{"osd":4,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.92500000000000004,"15min":0.93200000000000005},"min":{"1min":0.65300000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0620000000000001,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":1.044},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.88900000000000001,"15min":0.91300000000000003},"min":{"1min":0.60399999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.94299999999999995,"5min":1.484,"15min":1.484},"last":0.95399999999999996}]},{"osd":5,"last update":"Mon Sep 16 07:23:57 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.90800000000000003,"15min":0.92000000000000004},"min":{"1min":0.67800000000000005,"5min":0.441,"15min":0.441},"max":{"1min":1.1200000000000001,"5min":1.393,"15min":1.393},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.997,"15min":1.038},"min":{"1min":0.76200000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.234,"5min":2.028,"15min":2.028},"last":0.98299999999999998}]}]},{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":4747264,"data_stored":2093575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.74399999999999999,"15min":0.72299999999999998},"min":{"1min":0.57899999999999996,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98799999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72899999999999998,"15min":0.745},"min":{"1min":0.59199999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.026,"5min":1.036,"15min":1.036},"last":0.77000000000000002}]},{"osd":1,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.82399999999999995,"15min":0.80400000000000005},"min":{"1min":0.73199999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.97699999999999998,"5min":1.254,"15min":1.254},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.86599999999999999,"15min":0.84799999999999998},"min":{"1min":0.68300000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.1020000000000001,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.748}]},{"osd":2,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85499999999999998,"15min":0.82999999999999996},"min":{"1min":0.59099999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.98799999999999999,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.879},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.65400000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96299999999999997,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.96299999999999997}]},{"osd":3,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.70299999999999996,"15min":0.70899999999999996},"min":{"1min":0.53000000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.85599999999999998,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.77700000000000002,"15min":0.78800000000000003},"min":{"1min":0.44,"5min":0.376,"15min":0.376},"max":{"1min":1.131,"5min":1.165,"15min":1.165},"last":0.621}]},{"osd":4,"last update":"Mon Sep 16 07:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.82699999999999996,"15min":0.81000000000000005},"min":{"1min":0.497,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1759999999999999,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.82099999999999995},"min":{"1min":0.53900000000000003,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0700000000000001,"5min":2.569,"15min":2.569},"last":0.71199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:25:02.173 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T07:25:02.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient: get_monmap_and_config 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient: build_initial_monmap 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient: monmap: 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:02.280218+0000 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:02.280218+0000 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:02.281 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:02.282 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:02.282 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:02.282 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:02.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 1 Processor -- start 2024-09-16T07:25:02.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 1 -- start start 2024-09-16T07:25:02.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient: init 2024-09-16T07:25:02.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a0e90 0x7f42180a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient(hunting): picked mon.noname-b con 0x7f42180a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42180a1270 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f42180a2d50 0x7f42180a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient(hunting): picked mon.noname-a con 0x7f42180a2d50 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42180a36c0 con 0x7f42180a1270 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f421806cf50 con 0x7f42180a2d50 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421809f810 con 0x7f42180a0e90 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:02.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient(hunting): authenticate will time out at 1222.766290s 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f42180a2d50 0x7f42180a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d0d4700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a0e90 0x7f42180a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421c8d3700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f42180a1270 0x7f42180a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54432/0 (socket says 172.21.15.37:54432) 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421c8d3700 1 -- 172.21.15.37:0/450722084 learned_addr learned my addr 172.21.15.37:0/450722084 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f42180a2d50 0x7f42180a3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39356/0 (socket says 172.21.15.37:39356) 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 10 monclient(hunting): get_auth_request con 0x7f42180a2d50 auth_method 0 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d0d4700 10 monclient(hunting): get_auth_request con 0x7f42180a0e90 auth_method 0 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d0d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d0d4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:02.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d0d4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d0d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d0d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d0d4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 10 monclient(hunting): handle_auth_done global_id 54156 payload 274 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 1 -- 172.21.15.37:0/450722084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a0e90 msgr2=0x7f42180a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 1 --2- 172.21.15.37:0/450722084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a0e90 0x7f42180a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 1 -- 172.21.15.37:0/450722084 >> v1:172.21.15.37:6789/0 conn(0x7f42180a1270 legacy=0x7f42180a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 10 monclient: _finish_hunting 0 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 1 monclient: found mon.noname-a 2024-09-16T07:25:02.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 1 -- 172.21.15.37:0/450722084 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421809f250 con 0x7f42180a2d50 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421d8d5700 1 --2- 172.21.15.37:0/450722084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a2d50 0x7f42180a3180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4210009d90 tx=0x7f421000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95aacf97abd7e88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 1 -- 172.21.15.37:0/450722084 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421000e070 con 0x7f42180a2d50 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 10 monclient: dump: 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:02.289 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 10 monclient: _finish_auth 0 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 10 monclient: _check_auth_tickets 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 5 monclient: authenticate success, global_id 54156 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 1 -- 172.21.15.37:0/450722084 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421000bb20 con 0x7f42180a2d50 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 1 -- 172.21.15.37:0/450722084 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210012430 con 0x7f42180a2d50 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f421e0d6700 4 set_mon_vals no callback set 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.278+0000 7f420ffff700 10 monclient: dump: 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 10 monclient: shutdown 2024-09-16T07:25:02.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:02.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 1 -- 172.21.15.37:0/450722084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a2d50 msgr2=0x7f42180a3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:02.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 1 --2- 172.21.15.37:0/450722084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a2d50 0x7f42180a3180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4210009d90 tx=0x7f421000b330 comp rx=0 tx=0).stop 2024-09-16T07:25:02.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 1 -- 172.21.15.37:0/450722084 shutdown_connections 2024-09-16T07:25:02.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 1 --2- 172.21.15.37:0/450722084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a2d50 0x7f42180a3180 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 1 --2- 172.21.15.37:0/450722084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a0e90 0x7f42180a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 1 -- 172.21.15.37:0/450722084 >> 172.21.15.37:0/450722084 conn(0x7f42180a4740 msgr2=0x7f42180a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:02.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 1 -- 172.21.15.37:0/450722084 shutdown_connections 2024-09-16T07:25:02.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 1 -- 172.21.15.37:0/450722084 wait complete. 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 10 monclient: build_initial_monmap 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 10 monclient: monmap: 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:02.297 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 1 Processor -- start 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 1 -- start start 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.282+0000 7f421e0d6700 10 monclient: init 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a0e90 0x7f421813b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient(hunting): picked mon.noname-a con 0x7f42180a0e90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a1270 0x7f421813bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient(hunting): picked mon.noname-c con 0x7f42180a1270 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42180a2d50 0x7f421813c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient(hunting): picked mon.noname-b con 0x7f42180a2d50 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:02.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42181260d0 con 0x7f42180a0e90 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4218126c40 con 0x7f42180a2d50 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4218125270 con 0x7f42180a1270 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient(hunting): authenticate will time out at 1222.772749s 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a1270 0x7f421813bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a0e90 0x7f421813b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a0e90 0x7f421813b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39372/0 (socket says 172.21.15.37:39372) 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 1 -- 172.21.15.37:0/1723460321 learned_addr learned my addr 172.21.15.37:0/1723460321 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a1270 0x7f421813bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34952/0 (socket says 172.21.15.37:34952) 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 10 monclient(hunting): get_auth_request con 0x7f42180a0e90 auth_method 0 2024-09-16T07:25:02.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 10 monclient(hunting): get_auth_request con 0x7f42180a1270 auth_method 0 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d8d5700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42180a2d50 0x7f421813c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:02.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d8d5700 10 monclient(hunting): get_auth_request con 0x7f42180a2d50 auth_method 0 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d8d5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d8d5700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d8d5700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d8d5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d8d5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d8d5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 10 monclient(hunting): handle_auth_done global_id 44552 payload 824 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 1 -- 172.21.15.37:0/1723460321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42180a2d50 msgr2=0x7f421813c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42180a2d50 0x7f421813c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 1 -- 172.21.15.37:0/1723460321 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a0e90 msgr2=0x7f421813b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a0e90 0x7f421813b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 10 monclient: _finish_hunting 0 2024-09-16T07:25:02.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 1 monclient: found mon.noname-c 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 1 -- 172.21.15.37:0/1723460321 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4210009a70 con 0x7f42180a1270 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421d0d4700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a0e90 0x7f421813b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421c8d3700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a1270 0x7f421813bec0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f42180a0aa0 tx=0x7f421400c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 1 -- 172.21.15.37:0/1723460321 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4214005b60 con 0x7f42180a1270 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 10 monclient: dump: 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:02.302 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:02.303 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:02.303 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:02.303 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:02.303 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:02.303 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:02.303 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:02.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 10 monclient: _finish_auth 0 2024-09-16T07:25:02.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 10 monclient: _check_auth_tickets 2024-09-16T07:25:02.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:02.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 5 monclient: authenticate success, global_id 44552 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 1 -- 172.21.15.37:0/1723460321 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4214005cc0 con 0x7f42180a1270 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient: _renew_subs 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 1 -- 172.21.15.37:0/1723460321 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4214007310 con 0x7f42180a1270 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 1 -- 172.21.15.37:0/1723460321 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421806d780 con 0x7f42180a1270 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420f7fe700 4 set_mon_vals no callback set 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 10 monclient: dump: 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:02.304 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient: _renew_subs 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 1 -- 172.21.15.37:0/1723460321 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42181256b0 con 0x7f42180a1270 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:02.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:02.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f420dffb700 1 -- 172.21.15.37:0/1723460321 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f4214007680 con 0x7f42180a1270 2024-09-16T07:25:02.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.286+0000 7f421e0d6700 1 -- 172.21.15.37:0/1723460321 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42180a04d0 con 0x7f42180a1270 2024-09-16T07:25:02.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.290+0000 7f420dffb700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41fc035600 0x7f41fc037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:02.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.290+0000 7f420dffb700 1 -- 172.21.15.37:0/1723460321 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f4214046ef0 con 0x7f42180a1270 2024-09-16T07:25:02.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.290+0000 7f421d0d4700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41fc035600 0x7f41fc037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:02.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.290+0000 7f421d0d4700 10 monclient: get_auth_request con 0x7f41fc035600 auth_method 0 2024-09-16T07:25:02.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.290+0000 7f421d0d4700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41fc035600 0x7f41fc037ac0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f42080113d0 tx=0x7f4208009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:02.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.294+0000 7f420dffb700 1 -- 172.21.15.37:0/1723460321 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4214019730 con 0x7f42180a1270 2024-09-16T07:25:02.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.294+0000 7f420dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:02.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.294+0000 7f420dffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:02.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.466+0000 7f421e0d6700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:25:02.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.466+0000 7f421e0d6700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:25:02.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.466+0000 7f421e0d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:02.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.466+0000 7f421e0d6700 1 -- 172.21.15.37:0/1723460321 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4218126d80 con 0x7f42180a1270 2024-09-16T07:25:02.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.466+0000 7f420dffb700 1 -- 172.21.15.37:0/1723460321 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v315) v1 ==== 74+0+12199 (secure 0 0 0) 0x7f4218126d80 con 0x7f42180a1270 2024-09-16T07:25:02.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.466+0000 7f420dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T07:25:02.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.466+0000 7f420dffb700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:25:02.471 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:25:02.471 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":315,"fsid":"2b3c1781-6e4a-42f3-a6cb-6f1008258949","created":"2024-09-16T07:18:36.646941+0000","modified":"2024-09-16T07:24:40.671901+0000","last_up_change":"2024-09-16T07:18:46.372944+0000","last_in_change":"2024-09-16T07:18:38.297482+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T07:18:46.507362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T07:18:53.543683+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd764fc8-b5fb-40fe-a64f-ce7b8c1df525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6801","nonce":3137250735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6803","nonce":3137250735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6807","nonce":3137250735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3137250735},{"type":"v1","addr":"172.21.15.37:6805","nonce":3137250735}]},"public_addr":"172.21.15.37:6801/3137250735","cluster_addr":"172.21.15.37:6803/3137250735","heartbeat_back_addr":"172.21.15.37:6807/3137250735","heartbeat_front_addr":"172.21.15.37:6805/3137250735","state":["exists","up"]},{"osd":1,"uuid":"52d577a5-08b7-452d-aa35-e1560215a09e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6809","nonce":810396637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6811","nonce":810396637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6815","nonce":810396637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":810396637},{"type":"v1","addr":"172.21.15.37:6813","nonce":810396637}]},"public_addr":"172.21.15.37:6809/810396637","cluster_addr":"172.21.15.37:6811/810396637","heartbeat_back_addr":"172.21.15.37:6815/810396637","heartbeat_front_addr":"172.21.15.37:6813/810396637","state":["exists","up"]},{"osd":2,"uuid":"e3deb985-fb4e-42ee-8ed6-e6318cbca322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6817","nonce":265940208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6819","nonce":265940208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6823","nonce":265940208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":265940208},{"type":"v1","addr":"172.21.15.37:6821","nonce":265940208}]},"public_addr":"172.21.15.37:6817/265940208","cluster_addr":"172.21.15.37:6819/265940208","heartbeat_back_addr":"172.21.15.37:6823/265940208","heartbeat_front_addr":"172.21.15.37:6821/265940208","state":["exists","up"]},{"osd":3,"uuid":"ab447540-8ed1-4a0c-8f6c-dfeea8af02ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6809","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6811","nonce":2588720019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6813","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6815","nonce":2588720019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6823","nonce":2588720019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":2588720019},{"type":"v1","addr":"172.21.15.143:6819","nonce":2588720019}]},"public_addr":"172.21.15.143:6811/2588720019","cluster_addr":"172.21.15.143:6815/2588720019","heartbeat_back_addr":"172.21.15.143:6823/2588720019","heartbeat_front_addr":"172.21.15.143:6819/2588720019","state":["exists","up"]},{"osd":4,"uuid":"be421fc0-1edf-4f20-b530-c9cf4eb20a99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6810","nonce":3355421705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6814","nonce":3355421705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6822","nonce":3355421705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3355421705},{"type":"v1","addr":"172.21.15.143:6818","nonce":3355421705}]},"public_addr":"172.21.15.143:6810/3355421705","cluster_addr":"172.21.15.143:6814/3355421705","heartbeat_back_addr":"172.21.15.143:6822/3355421705","heartbeat_front_addr":"172.21.15.143:6818/3355421705","state":["exists","up"]},{"osd":5,"uuid":"8c7725bd-5f56-4fa0-b455-6569e313f9fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6801","nonce":3921632375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6803","nonce":3921632375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6807","nonce":3921632375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3921632375},{"type":"v1","addr":"172.21.15.143:6805","nonce":3921632375}]},"public_addr":"172.21.15.143:6801/3921632375","cluster_addr":"172.21.15.143:6803/3921632375","heartbeat_back_addr":"172.21.15.143:6807/3921632375","heartbeat_front_addr":"172.21.15.143:6805/3921632375","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.492199+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.565060+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.519201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.777218+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.842229+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T07:18:43.307398+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T07:25:02.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 -- 172.21.15.37:0/1723460321 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41fc035600 msgr2=0x7f41fc037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:02.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41fc035600 0x7f41fc037ac0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f42080113d0 tx=0x7f4208009240 comp rx=0 tx=0).stop 2024-09-16T07:25:02.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 10 monclient: shutdown 2024-09-16T07:25:02.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:02.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 -- 172.21.15.37:0/1723460321 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a1270 msgr2=0x7f421813bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:02.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a1270 0x7f421813bec0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f42180a0aa0 tx=0x7f421400c6e0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 -- 172.21.15.37:0/1723460321 shutdown_connections 2024-09-16T07:25:02.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42180a2d50 0x7f421813c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f41fc035600 0x7f41fc037ac0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42180a1270 0x7f421813bec0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 --2- 172.21.15.37:0/1723460321 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42180a0e90 0x7f421813b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:02.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 -- 172.21.15.37:0/1723460321 >> 172.21.15.37:0/1723460321 conn(0x7f42180a4740 msgr2=0x7f421809e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:02.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 -- 172.21.15.37:0/1723460321 shutdown_connections 2024-09-16T07:25:02.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:02.470+0000 7f421e0d6700 1 -- 172.21.15.37:0/1723460321 wait complete. 2024-09-16T07:25:03.486 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T07:25:03.487 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T07:25:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient: get_monmap_and_config 2024-09-16T07:25:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient: build_initial_monmap 2024-09-16T07:25:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient: monmap: 2024-09-16T07:25:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:03.616654+0000 2024-09-16T07:25:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:03.616654+0000 2024-09-16T07:25:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:03.618 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 1 Processor -- start 2024-09-16T07:25:03.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 1 -- start start 2024-09-16T07:25:03.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient: init 2024-09-16T07:25:03.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:03.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T07:25:03.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9500a2750 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a3240 0x7fd9500a0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient(hunting): picked mon.noname-b con 0x7fd9500a3240 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd95013a9d0 0x7fd95013cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient(hunting): picked mon.noname-c con 0x7fd95013a9d0 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95006ce50 con 0x7fd9500a2750 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95009f7e0 con 0x7fd95013a9d0 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9500a3980 con 0x7fd9500a3240 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd9557fe700 10 monclient(hunting): authenticate will time out at 1224.102911s 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94ffff700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd9500a2750 0x7fd9500a2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54434/0 (socket says 172.21.15.37:54434) 2024-09-16T07:25:03.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94ffff700 1 -- 172.21.15.37:0/946614362 learned_addr learned my addr 172.21.15.37:0/946614362 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 1 --2- 172.21.15.37:0/946614362 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a3240 0x7fd9500a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 10 monclient(hunting): get_auth_request con 0x7fd9500a3240 auth_method 0 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd954ffd700 1 --2- 172.21.15.37:0/946614362 >> v2:172.21.15.143:3300/0 conn(0x7fd95013a9d0 0x7fd95013cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd954ffd700 10 monclient(hunting): get_auth_request con 0x7fd95013a9d0 auth_method 0 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd954ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:03.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd954ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd954ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd954ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd954ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd954ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94effd700 1 -- 172.21.15.37:0/946614362 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389239435 0 0) 0x7fd95006ce50 con 0x7fd9500a2750 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94effd700 10 monclient(hunting): _negotiate 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94effd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94effd700 10 monclient(hunting): my global_id is 54165 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94effd700 1 -- 172.21.15.37:0/946614362 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9380035d0 con 0x7fd9500a2750 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 10 monclient(hunting): handle_auth_done global_id 44555 payload 274 2024-09-16T07:25:03.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 1 -- 172.21.15.37:0/946614362 >> v2:172.21.15.143:3300/0 conn(0x7fd95013a9d0 msgr2=0x7fd95013cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 1 --2- 172.21.15.37:0/946614362 >> v2:172.21.15.143:3300/0 conn(0x7fd95013a9d0 0x7fd95013cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 1 -- 172.21.15.37:0/946614362 >> v1:172.21.15.37:6789/0 conn(0x7fd9500a2750 legacy=0x7fd9500a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 1 monclient: found mon.noname-b 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.614+0000 7fd94f7fe700 1 -- 172.21.15.37:0/946614362 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95006d0b0 con 0x7fd9500a3240 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94f7fe700 1 --2- 172.21.15.37:0/946614362 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a3240 0x7fd9500a0780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd940009850 tx=0x7fd940009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be953fa2f6b79ecc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 1 -- 172.21.15.37:0/946614362 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940011830 con 0x7fd9500a3240 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 10 monclient: dump: 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:03.630 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 10 monclient: _finish_auth 0 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 10 monclient: _check_auth_tickets 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:03.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 5 monclient: authenticate success, global_id 44555 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 1 -- 172.21.15.37:0/946614362 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd940011990 con 0x7fd9500a3240 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 4 set_mon_vals no callback set 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 1 -- 172.21.15.37:0/946614362 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940011d40 con 0x7fd9500a3240 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94effd700 10 monclient: dump: 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.632 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient: shutdown 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 -- 172.21.15.37:0/946614362 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a3240 msgr2=0x7fd9500a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 --2- 172.21.15.37:0/946614362 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a3240 0x7fd9500a0780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd940009850 tx=0x7fd940009b60 comp rx=0 tx=0).stop 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 -- 172.21.15.37:0/946614362 shutdown_connections 2024-09-16T07:25:03.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 --2- 172.21.15.37:0/946614362 >> v2:172.21.15.143:3300/0 conn(0x7fd95013a9d0 0x7fd95013cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 --2- 172.21.15.37:0/946614362 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a3240 0x7fd9500a0780 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 -- 172.21.15.37:0/946614362 >> 172.21.15.37:0/946614362 conn(0x7fd9500a4850 msgr2=0x7fd9500a4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 -- 172.21.15.37:0/946614362 shutdown_connections 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 -- 172.21.15.37:0/946614362 wait complete. 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient: build_initial_monmap 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient: monmap: 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:03.634 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 Processor -- start 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 -- start start 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient: init 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a2750 0x7fd9501ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient(hunting): picked mon.noname-c con 0x7fd9500a2750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9500a3240 0x7fd9501ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient(hunting): picked mon.noname-a con 0x7fd9500a3240 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd95013a9d0 0x7fd95009c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient(hunting): picked mon.noname-b con 0x7fd95013a9d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95013cad0 con 0x7fd9500a3240 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95009f0b0 con 0x7fd95013a9d0 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95011fb40 con 0x7fd9500a2750 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd9557fe700 10 monclient(hunting): authenticate will time out at 1224.107452s 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a2750 0x7fd9501ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a2750 0x7fd9501ca480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34972/0 (socket says 172.21.15.37:34972) 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94ffff700 1 -- 172.21.15.37:0/3183097278 learned_addr learned my addr 172.21.15.37:0/3183097278 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94ffff700 10 monclient(hunting): get_auth_request con 0x7fd9500a2750 auth_method 0 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94f7fe700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9500a3240 0x7fd9501ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd954ffd700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd95013a9d0 0x7fd95009c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd954ffd700 10 monclient(hunting): get_auth_request con 0x7fd95013a9d0 auth_method 0 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd954ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd954ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd954ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94f7fe700 10 monclient(hunting): get_auth_request con 0x7fd9500a3240 auth_method 0 2024-09-16T07:25:03.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.618+0000 7fd94ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd954ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd954ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd954ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 10 monclient(hunting): handle_auth_done global_id 44558 payload 824 2024-09-16T07:25:03.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 1 -- 172.21.15.37:0/3183097278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd95013a9d0 msgr2=0x7fd95009c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd95013a9d0 0x7fd95009c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 1 -- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9500a3240 msgr2=0x7fd9501ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9500a3240 0x7fd9501ca9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 10 monclient: _finish_hunting 0 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 1 monclient: found mon.noname-c 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 1 -- 172.21.15.37:0/3183097278 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9400094b0 con 0x7fd9500a2750 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94ffff700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a2750 0x7fd9501ca480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd94400c700 tx=0x7fd94400ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 1 -- 172.21.15.37:0/3183097278 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd944005d90 con 0x7fd9500a2750 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:03.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 10 monclient: dump: 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 10 monclient: _finish_auth 0 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 10 monclient: _check_auth_tickets 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd9557fe700 5 monclient: authenticate success, global_id 44558 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd9557fe700 10 monclient: _renew_subs 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd9557fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95011fe20 con 0x7fd9500a2750 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 1 -- 172.21.15.37:0/3183097278 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9440070d0 con 0x7fd9500a2750 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 1 -- 172.21.15.37:0/3183097278 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd944007480 con 0x7fd9500a2750 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 10 monclient: dump: 2024-09-16T07:25:03.641 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd9557fe700 10 monclient: _renew_subs 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd9557fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95009e2c0 con 0x7fd9500a2750 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94e7fc700 4 set_mon_vals no callback set 2024-09-16T07:25:03.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd9557fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:03.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd95013d300 con 0x7fd9500a2750 2024-09-16T07:25:03.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 1 -- 172.21.15.37:0/3183097278 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fd9440075e0 con 0x7fd9500a2750 2024-09-16T07:25:03.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd93c035600 0x7fd93c037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 1 -- 172.21.15.37:0/3183097278 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fd944046eb0 con 0x7fd9500a2750 2024-09-16T07:25:03.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x7fd93c03b410 0x7fd93c03d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 1 -- 172.21.15.37:0/3183097278 --> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd93c03dee0 con 0x7fd93c03b410 2024-09-16T07:25:03.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 1 -- 172.21.15.37:0/3183097278 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd944018260 con 0x7fd9500a2750 2024-09-16T07:25:03.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94cff9700 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T07:25:03.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94f7fe700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd93c035600 0x7fd93c037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd954ffd700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x7fd93c03b410 0x7fd93c03d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd94f7fe700 10 monclient: get_auth_request con 0x7fd93c035600 auth_method 0 2024-09-16T07:25:03.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.622+0000 7fd954ffd700 10 monclient: get_auth_request con 0x7fd93c03b410 auth_method 0 2024-09-16T07:25:03.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.626+0000 7fd954ffd700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x7fd93c03b410 0x7fd93c03d830 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:03.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.626+0000 7fd94f7fe700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd93c035600 0x7fd93c037ac0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd9400090b0 tx=0x7fd940008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:03.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.626+0000 7fd94cff9700 1 -- 172.21.15.37:0/3183097278 <== osd.0 v2:172.21.15.37:6800/3137250735 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fd93c03dee0 con 0x7fd93c03b410 2024-09-16T07:25:03.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 --> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd95011f580 con 0x7fd93c03b410 2024-09-16T07:25:03.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd94cff9700 1 -- 172.21.15.37:0/3183097278 <== osd.0 v2:172.21.15.37:6800/3137250735 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd95011f580 con 0x7fd93c03b410 2024-09-16T07:25:03.646 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:25:03.646 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T07:25:03.647 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:25:03.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x7fd93c03b410 msgr2=0x7fd93c03d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:6800/3137250735,v1:172.21.15.37:6801/3137250735] conn(0x7fd93c03b410 0x7fd93c03d830 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd93c035600 msgr2=0x7fd93c037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fd93c035600 0x7fd93c037ac0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd9400090b0 tx=0x7fd940008030 comp rx=0 tx=0).stop 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 10 monclient: shutdown 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a2750 msgr2=0x7fd9501ca480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 --2- 172.21.15.37:0/3183097278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9500a2750 0x7fd9501ca480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd94400c700 tx=0x7fd94400ca10 comp rx=0 tx=0).stop 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd954ffd700 1 -- 172.21.15.37:0/3183097278 reap_dead start 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 shutdown_connections 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 >> 172.21.15.37:0/3183097278 conn(0x7fd9500a4850 msgr2=0x7fd9500a6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:03.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 shutdown_connections 2024-09-16T07:25:03.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.642+0000 7fd9557fe700 1 -- 172.21.15.37:0/3183097278 wait complete. 2024-09-16T07:25:03.661 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T07:25:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 10 monclient: get_monmap_and_config 2024-09-16T07:25:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 10 monclient: build_initial_monmap 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 10 monclient: monmap: 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:03.785683+0000 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:03.785683+0000 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:03.788 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 1 Processor -- start 2024-09-16T07:25:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 1 -- start start 2024-09-16T07:25:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 10 monclient: init 2024-09-16T07:25:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-16T07:25:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6a000a2750 0x7f6a000a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 10 monclient(hunting): picked mon.noname-a con 0x7f6a000a2750 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a000a3070 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0013b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 10 monclient(hunting): picked mon.noname-b con 0x7f6a00133730 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a00099be0 con 0x7f6a000a3070 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.782+0000 7f6a067ad700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0009d840 con 0x7f6a000a2750 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0009de40 con 0x7f6a00133730 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 10 monclient(hunting): authenticate will time out at 1224.272074s 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6a000a2750 0x7f6a000a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6a000a2750 0x7f6a000a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39394/0 (socket says 172.21.15.37:39394) 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a04faa700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6a000a3070 0x7f6a00133020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54436/0 (socket says 172.21.15.37:54436) 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0013b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 1 -- 172.21.15.37:0/3904325443 learned_addr learned my addr 172.21.15.37:0/3904325443 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 10 monclient(hunting): get_auth_request con 0x7f6a000a2750 auth_method 0 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 10 monclient(hunting): get_auth_request con 0x7f6a00133730 auth_method 0 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 10 monclient(hunting): handle_auth_done global_id 44561 payload 274 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 1 -- 172.21.15.37:0/3904325443 >> v2:172.21.15.37:3300/0 conn(0x7f6a000a2750 msgr2=0x7f6a000a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 1 --2- 172.21.15.37:0/3904325443 >> v2:172.21.15.37:3300/0 conn(0x7f6a000a2750 0x7f6a000a2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 1 -- 172.21.15.37:0/3904325443 >> v1:172.21.15.37:6789/0 conn(0x7f6a000a3070 legacy=0x7f6a00133020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 10 monclient: _finish_hunting 0 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 1 monclient: found mon.noname-b 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 1 -- 172.21.15.37:0/3904325443 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957315438 0 0) 0x7f6a00099be0 con 0x7f6a000a3070 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 1 -- 172.21.15.37:0/3904325443 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0009d530 con 0x7f6a00133730 2024-09-16T07:25:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a057ab700 1 --2- 172.21.15.37:0/3904325443 >> v2:172.21.15.37:3300/0 conn(0x7f6a000a2750 0x7f6a000a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a05fac700 1 --2- 172.21.15.37:0/3904325443 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0013b310 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f69fc009850 tx=0x7f69fc009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65d36093998715d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 1 -- 172.21.15.37:0/3904325443 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc011830 con 0x7f6a00133730 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 10 monclient: dump: 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 10 monclient: _finish_auth 0 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 10 monclient: _check_auth_tickets 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 5 monclient: authenticate success, global_id 44561 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 1 -- 172.21.15.37:0/3904325443 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc011990 con 0x7f6a00133730 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 4 set_mon_vals no callback set 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 1 -- 172.21.15.37:0/3904325443 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc011d40 con 0x7f6a00133730 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f69f7fff700 10 monclient: dump: 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 10 monclient: shutdown 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 1 -- 172.21.15.37:0/3904325443 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 msgr2=0x7f6a0013b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 1 --2- 172.21.15.37:0/3904325443 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0013b310 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f69fc009850 tx=0x7f69fc009b60 comp rx=0 tx=0).stop 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 1 -- 172.21.15.37:0/3904325443 shutdown_connections 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 1 --2- 172.21.15.37:0/3904325443 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0013b310 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 1 --2- 172.21.15.37:0/3904325443 >> v2:172.21.15.37:3300/0 conn(0x7f6a000a2750 0x7f6a000a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 1 -- 172.21.15.37:0/3904325443 >> 172.21.15.37:0/3904325443 conn(0x7f6a000a3ea0 msgr2=0x7f6a000a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 1 -- 172.21.15.37:0/3904325443 shutdown_connections 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 1 -- 172.21.15.37:0/3904325443 wait complete. 2024-09-16T07:25:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 10 monclient: build_initial_monmap 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.786+0000 7f6a067ad700 10 monclient: monmap: 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 1 Processor -- start 2024-09-16T07:25:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 1 -- start start 2024-09-16T07:25:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient: init 2024-09-16T07:25:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:25:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a000a2750 0x7f6a001c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient(hunting): picked mon.noname-a con 0x7f6a000a2750 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a000a3070 0x7f6a00132580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient(hunting): picked mon.noname-b con 0x7f6a000a3070 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0011bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient(hunting): picked mon.noname-c con 0x7f6a00133730 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0011fa60 con 0x7f6a000a2750 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0011ff40 con 0x7f6a000a3070 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0006cf00 con 0x7f6a00133730 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient(hunting): authenticate will time out at 1224.277145s 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0011bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a057ab700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a000a2750 0x7f6a001c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0011bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34984/0 (socket says 172.21.15.37:34984) 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 1 -- 172.21.15.37:0/49485721 learned_addr learned my addr 172.21.15.37:0/49485721 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 10 monclient(hunting): get_auth_request con 0x7f6a00133730 auth_method 0 2024-09-16T07:25:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a057ab700 10 monclient(hunting): get_auth_request con 0x7f6a000a2750 auth_method 0 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a057ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a057ab700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a057ab700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a04faa700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a000a3070 0x7f6a00132580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a057ab700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a057ab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a057ab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a04faa700 10 monclient(hunting): get_auth_request con 0x7f6a000a3070 auth_method 0 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a04faa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a04faa700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a04faa700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a04faa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a04faa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a04faa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 10 monclient(hunting): handle_auth_done global_id 44564 payload 824 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 1 -- 172.21.15.37:0/49485721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a000a3070 msgr2=0x7f6a00132580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a000a3070 0x7f6a00132580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 1 -- 172.21.15.37:0/49485721 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a000a2750 msgr2=0x7f6a001c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a000a2750 0x7f6a001c8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 10 monclient: _finish_hunting 0 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 1 monclient: found mon.noname-c 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 1 -- 172.21.15.37:0/49485721 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f8009b30 con 0x7f6a00133730 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a05fac700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0011bf90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0090b0 tx=0x7f69fc004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 1 -- 172.21.15.37:0/49485721 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc01a8d0 con 0x7f6a00133730 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 10 monclient: dump: 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 10 monclient: _finish_auth 0 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 1 -- 172.21.15.37:0/49485721 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc01f070 con 0x7f6a00133730 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 5 monclient: authenticate success, global_id 44564 2024-09-16T07:25:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient: _renew_subs 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 1 -- 172.21.15.37:0/49485721 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69fc0094b0 con 0x7f6a00133730 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 1 -- 172.21.15.37:0/49485721 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc01ac70 con 0x7f6a00133730 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 10 monclient: dump: 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f77fe700 4 set_mon_vals no callback set 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient: _renew_subs 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f6a067ad700 1 -- 172.21.15.37:0/49485721 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a0011e530 con 0x7f6a00133730 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.790+0000 7f69f5ffb700 1 -- 172.21.15.37:0/49485721 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f69fc01b430 con 0x7f6a00133730 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.794+0000 7f69f5ffb700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f69e4035600 0x7f69e4037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.794+0000 7f69f5ffb700 1 -- 172.21.15.37:0/49485721 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f69fc048b10 con 0x7f6a00133730 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.794+0000 7f6a057ab700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f69e4035600 0x7f69e4037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.794+0000 7f6a057ab700 10 monclient: get_auth_request con 0x7f69e4035600 auth_method 0 2024-09-16T07:25:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.794+0000 7f6a067ad700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.794+0000 7f6a067ad700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.794+0000 7f6a067ad700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.794+0000 7f6a067ad700 1 -- 172.21.15.37:0/49485721 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0009cd90 con 0x7f6a00133730 2024-09-16T07:25:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.794+0000 7f6a057ab700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f69e4035600 0x7f69e4037ac0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f69f80072c0 tx=0x7f69f800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.798+0000 7f69f5ffb700 1 -- 172.21.15.37:0/49485721 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f69fc015080 con 0x7f6a00133730 2024-09-16T07:25:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.798+0000 7f69f5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.798+0000 7f69f5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:03.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.974+0000 7f6a067ad700 1 -- 172.21.15.37:0/49485721 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a00133570 con 0x7f69e4035600 2024-09-16T07:25:03.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.974+0000 7f69f5ffb700 1 -- 172.21.15.37:0/49485721 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6a00133570 con 0x7f69e4035600 2024-09-16T07:25:03.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 -- 172.21.15.37:0/49485721 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f69e4035600 msgr2=0x7f69e4037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f69e4035600 0x7f69e4037ac0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f69f80072c0 tx=0x7f69f800b040 comp rx=0 tx=0).stop 2024-09-16T07:25:03.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 10 monclient: shutdown 2024-09-16T07:25:03.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:03.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 -- 172.21.15.37:0/49485721 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 msgr2=0x7f6a0011bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:03.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0011bf90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0090b0 tx=0x7f69fc004770 comp rx=0 tx=0).stop 2024-09-16T07:25:03.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 -- 172.21.15.37:0/49485721 shutdown_connections 2024-09-16T07:25:03.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f69e4035600 0x7f69e4037ac0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a00133730 0x7f6a0011bf90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a000a3070 0x7f6a00132580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 --2- 172.21.15.37:0/49485721 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a000a2750 0x7f6a001c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:03.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 -- 172.21.15.37:0/49485721 >> 172.21.15.37:0/49485721 conn(0x7f6a000a3ea0 msgr2=0x7f6a0006f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:03.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 -- 172.21.15.37:0/49485721 shutdown_connections 2024-09-16T07:25:03.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:03.978+0000 7f6a067ad700 1 -- 172.21.15.37:0/49485721 wait complete. 2024-09-16T07:25:03.982 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T07:25:03.995 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T07:25:03.995 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T07:25:04.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.110+0000 7faac7fff700 10 monclient: get_monmap_and_config 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.110+0000 7faac7fff700 10 monclient: build_initial_monmap 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.110+0000 7faac7fff700 10 monclient: monmap: 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:04.114144+0000 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:04.114144+0000 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:04.115 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:04.116 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:04.116 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.110+0000 7faac7fff700 1 Processor -- start 2024-09-16T07:25:04.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.110+0000 7faac7fff700 1 -- start start 2024-09-16T07:25:04.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.110+0000 7faac7fff700 10 monclient: init 2024-09-16T07:25:04.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:04.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T07:25:04.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faac80a24d0 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:04.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faac806cb00 0x7faac813a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient(hunting): picked mon.noname-c con 0x7faac806cb00 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:04.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac80a2fc0 0x7faac813cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient(hunting): picked mon.noname-b con 0x7faac80a2fc0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:04.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac809a020 con 0x7faac80a24d0 2024-09-16T07:25:04.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faac80a4140 con 0x7faac806cb00 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac8099d40 con 0x7faac80a2fc0 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient(hunting): authenticate will time out at 1224.600580s 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac80a2fc0 0x7faac813cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac80a2fc0 0x7faac813cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34986/0 (socket says 172.21.15.37:34986) 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 1 -- 172.21.15.37:0/102342536 learned_addr learned my addr 172.21.15.37:0/102342536 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 10 monclient(hunting): get_auth_request con 0x7faac80a2fc0 auth_method 0 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac67fc700 1 --2- 172.21.15.37:0/102342536 >> v2:172.21.15.143:3300/0 conn(0x7faac806cb00 0x7faac813a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac67fc700 10 monclient(hunting): get_auth_request con 0x7faac806cb00 auth_method 0 2024-09-16T07:25:04.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 10 monclient(hunting): handle_auth_done global_id 44567 payload 274 2024-09-16T07:25:04.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 1 -- 172.21.15.37:0/102342536 >> v2:172.21.15.143:3300/0 conn(0x7faac806cb00 msgr2=0x7faac813a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 1 --2- 172.21.15.37:0/102342536 >> v2:172.21.15.143:3300/0 conn(0x7faac806cb00 0x7faac813a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 1 -- 172.21.15.37:0/102342536 >> v1:172.21.15.37:6789/0 conn(0x7faac80a24d0 legacy=0x7faac80a28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 10 monclient: _finish_hunting 0 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 1 monclient: found mon.noname-b 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 1 -- 172.21.15.37:0/102342536 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac809a300 con 0x7faac80a2fc0 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac77fe700 1 --2- 172.21.15.37:0/102342536 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac80a2fc0 0x7faac813cca0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faac0009e50 tx=0x7faac000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5103b08e4e6b4e64 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 1 -- 172.21.15.37:0/102342536 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac000e070 con 0x7faac80a2fc0 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:25:04.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 10 monclient: dump: 2024-09-16T07:25:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.132 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 10 monclient: _finish_auth 0 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 5 monclient: authenticate success, global_id 44567 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 1 -- 172.21.15.37:0/102342536 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac000b9c0 con 0x7faac80a2fc0 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 4 set_mon_vals no callback set 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 1 -- 172.21.15.37:0/102342536 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac000bd70 con 0x7faac80a2fc0 2024-09-16T07:25:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac5ffb700 10 monclient: dump: 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient: shutdown 2024-09-16T07:25:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 -- 172.21.15.37:0/102342536 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac80a2fc0 msgr2=0x7faac813cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 --2- 172.21.15.37:0/102342536 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac80a2fc0 0x7faac813cca0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faac0009e50 tx=0x7faac000b330 comp rx=0 tx=0).stop 2024-09-16T07:25:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 -- 172.21.15.37:0/102342536 shutdown_connections 2024-09-16T07:25:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 --2- 172.21.15.37:0/102342536 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac80a2fc0 0x7faac813cca0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 --2- 172.21.15.37:0/102342536 >> v2:172.21.15.143:3300/0 conn(0x7faac806cb00 0x7faac813a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 -- 172.21.15.37:0/102342536 >> 172.21.15.37:0/102342536 conn(0x7faac80a45d0 msgr2=0x7faac80a49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 -- 172.21.15.37:0/102342536 shutdown_connections 2024-09-16T07:25:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 1 -- 172.21.15.37:0/102342536 wait complete. 2024-09-16T07:25:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient: build_initial_monmap 2024-09-16T07:25:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.114+0000 7faac7fff700 10 monclient: monmap: 2024-09-16T07:25:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 1 Processor -- start 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 1 -- start start 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient: init 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac806cb00 0x7faac8122a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient(hunting): picked mon.noname-c con 0x7faac806cb00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faac80a24d0 0x7faac8122fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient(hunting): picked mon.noname-b con 0x7faac80a24d0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faac80a2fc0 0x7faac81234f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient(hunting): picked mon.noname-a con 0x7faac80a2fc0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:04.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac80a0340 con 0x7faac80a2fc0 2024-09-16T07:25:04.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac8126130 con 0x7faac80a24d0 2024-09-16T07:25:04.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac8135490 con 0x7faac806cb00 2024-09-16T07:25:04.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:04.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient(hunting): authenticate will time out at 1224.605038s 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faac80a2fc0 0x7faac81234f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac6ffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac806cb00 0x7faac8122a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faac80a2fc0 0x7faac81234f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39408/0 (socket says 172.21.15.37:39408) 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 1 -- 172.21.15.37:0/762152379 learned_addr learned my addr 172.21.15.37:0/762152379 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 10 monclient(hunting): get_auth_request con 0x7faac80a2fc0 auth_method 0 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac6ffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac806cb00 0x7faac8122a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34990/0 (socket says 172.21.15.37:34990) 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac6ffd700 10 monclient(hunting): get_auth_request con 0x7faac806cb00 auth_method 0 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac67fc700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faac80a24d0 0x7faac8122fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac67fc700 10 monclient(hunting): get_auth_request con 0x7faac80a24d0 auth_method 0 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac67fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 10 monclient(hunting): handle_auth_done global_id 54183 payload 824 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 1 -- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac806cb00 msgr2=0x7faac8122a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac806cb00 0x7faac8122a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 1 -- 172.21.15.37:0/762152379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faac80a24d0 msgr2=0x7faac8122fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faac80a24d0 0x7faac8122fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 10 monclient: _finish_hunting 0 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 1 monclient: found mon.noname-a 2024-09-16T07:25:04.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:04.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 1 -- 172.21.15.37:0/762152379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac0009b30 con 0x7faac80a2fc0 2024-09-16T07:25:04.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac6ffd700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faac806cb00 0x7faac8122a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:25:04.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac77fe700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faac80a2fc0 0x7faac81234f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faac000b730 tx=0x7faac0004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 1 -- 172.21.15.37:0/762152379 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac00048b0 con 0x7faac80a2fc0 2024-09-16T07:25:04.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:04.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 10 monclient: dump: 2024-09-16T07:25:04.142 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 10 monclient: _finish_auth 0 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:04.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 1 -- 172.21.15.37:0/762152379 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac00099b0 con 0x7faac80a2fc0 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 5 monclient: authenticate success, global_id 54183 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient: _renew_subs 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac809b7b0 con 0x7faac80a2fc0 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 1 -- 172.21.15.37:0/762152379 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac001c680 con 0x7faac80a2fc0 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 10 monclient: dump: 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.144 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac57fa700 4 set_mon_vals no callback set 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient: _renew_subs 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac8125e50 con 0x7faac80a2fc0 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.118+0000 7faaaf7fe700 1 -- 172.21.15.37:0/762152379 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7faac001c7e0 con 0x7faac80a2fc0 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faaaf7fe700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faab4035600 0x7faab4037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faaaf7fe700 1 -- 172.21.15.37:0/762152379 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7faac00495d0 con 0x7faac80a2fc0 2024-09-16T07:25:04.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faac7fff700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x7faac80612c0 0x7faac80616a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 --> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faac81262e0 con 0x7faac80612c0 2024-09-16T07:25:04.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faac67fc700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x7faac80612c0 0x7faac80616a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faac67fc700 10 monclient: get_auth_request con 0x7faac80612c0 auth_method 0 2024-09-16T07:25:04.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faac6ffd700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faab4035600 0x7faab4037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faac6ffd700 10 monclient: get_auth_request con 0x7faab4035600 auth_method 0 2024-09-16T07:25:04.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faac67fc700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x7faac80612c0 0x7faac80616a0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faac6ffd700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faab4035600 0x7faab4037ac0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7faabc001c90 tx=0x7faabc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.122+0000 7faaaf7fe700 1 -- 172.21.15.37:0/762152379 <== osd.1 v2:172.21.15.37:6808/810396637 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7faac81262e0 con 0x7faac80612c0 2024-09-16T07:25:04.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.138+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 --> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7faac80a28b0 con 0x7faac80612c0 2024-09-16T07:25:04.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.138+0000 7faaaf7fe700 1 -- 172.21.15.37:0/762152379 <== osd.1 v2:172.21.15.37:6808/810396637 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7faac80a28b0 con 0x7faac80612c0 2024-09-16T07:25:04.147 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:25:04.147 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T07:25:04.148 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x7faac80612c0 msgr2=0x7faac80616a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:6808/810396637,v1:172.21.15.37:6809/810396637] conn(0x7faac80612c0 0x7faac80616a0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faab4035600 msgr2=0x7faab4037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7faab4035600 0x7faab4037ac0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7faabc001c90 tx=0x7faabc009040 comp rx=0 tx=0).stop 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 10 monclient: shutdown 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faac80a2fc0 msgr2=0x7faac81234f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 1 --2- 172.21.15.37:0/762152379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faac80a2fc0 0x7faac81234f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faac000b730 tx=0x7faac0004af0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac77fe700 1 -- 172.21.15.37:0/762152379 reap_dead start 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 shutdown_connections 2024-09-16T07:25:04.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 >> 172.21.15.37:0/762152379 conn(0x7faac80a45d0 msgr2=0x7faac809fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:04.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 shutdown_connections 2024-09-16T07:25:04.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.142+0000 7faac7fff700 1 -- 172.21.15.37:0/762152379 wait complete. 2024-09-16T07:25:04.162 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T07:25:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.270+0000 7f7bf4c15700 10 monclient: get_monmap_and_config 2024-09-16T07:25:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.270+0000 7f7bf4c15700 10 monclient: build_initial_monmap 2024-09-16T07:25:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.270+0000 7f7bf4c15700 10 monclient: monmap: 2024-09-16T07:25:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.276 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:04.275320+0000 2024-09-16T07:25:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:04.275320+0000 2024-09-16T07:25:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:04.277 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:04.277 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 Processor -- start 2024-09-16T07:25:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 -- start start 2024-09-16T07:25:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient: init 2024-09-16T07:25:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T07:25:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7bf006a750 0x7f7bf006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient(hunting): picked mon.noname-c con 0x7f7bf006a750 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:04.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7bf006d8f0 0x7f7bf006b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient(hunting): picked mon.noname-a con 0x7f7bf006d8f0 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0135220 0x7f7bf0135650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient(hunting): picked mon.noname-b con 0x7f7bf0135220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf006d460 con 0x7f7bf006d8f0 2024-09-16T07:25:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf006b5b0 con 0x7f7bf006a750 2024-09-16T07:25:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf009ec20 con 0x7f7bf0135220 2024-09-16T07:25:04.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient(hunting): authenticate will time out at 1224.761526s 2024-09-16T07:25:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beeffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7bf006d8f0 0x7f7bf006b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0135220 0x7f7bf0135650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beeffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7bf006d8f0 0x7f7bf006b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39416/0 (socket says 172.21.15.37:39416) 2024-09-16T07:25:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beeffd700 1 -- 172.21.15.37:0/4227138644 learned_addr learned my addr 172.21.15.37:0/4227138644 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 10 monclient(hunting): get_auth_request con 0x7f7bf0135220 auth_method 0 2024-09-16T07:25:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beeffd700 10 monclient(hunting): get_auth_request con 0x7f7bf006d8f0 auth_method 0 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bef7fe700 1 --2- 172.21.15.37:0/4227138644 >> v2:172.21.15.143:3300/0 conn(0x7f7bf006a750 0x7f7bf006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bef7fe700 10 monclient(hunting): get_auth_request con 0x7f7bf006a750 auth_method 0 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 10 monclient(hunting): handle_auth_done global_id 44573 payload 274 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 1 -- 172.21.15.37:0/4227138644 >> v2:172.21.15.143:3300/0 conn(0x7f7bf006a750 msgr2=0x7f7bf006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 1 --2- 172.21.15.37:0/4227138644 >> v2:172.21.15.143:3300/0 conn(0x7f7bf006a750 0x7f7bf006ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 1 -- 172.21.15.37:0/4227138644 >> v2:172.21.15.37:3300/0 conn(0x7f7bf006d8f0 msgr2=0x7f7bf006b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 1 --2- 172.21.15.37:0/4227138644 >> v2:172.21.15.37:3300/0 conn(0x7f7bf006d8f0 0x7f7bf006b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 10 monclient: _finish_hunting 0 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 1 monclient: found mon.noname-b 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 1 -- 172.21.15.37:0/4227138644 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf009ef00 con 0x7f7bf0135220 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bef7fe700 1 --2- 172.21.15.37:0/4227138644 >> v2:172.21.15.143:3300/0 conn(0x7f7bf006a750 0x7f7bf006ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:25:04.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7beffff700 1 --2- 172.21.15.37:0/4227138644 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0135220 0x7f7bf0135650 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7bf006b7c0 tx=0x7f7be8009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d883c6c704d2b91a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 1 -- 172.21.15.37:0/4227138644 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8011830 con 0x7f7bf0135220 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 10 monclient: dump: 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.291 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 10 monclient: _finish_auth 0 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 5 monclient: authenticate success, global_id 44573 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 1 -- 172.21.15.37:0/4227138644 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8011990 con 0x7f7bf0135220 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 4 set_mon_vals no callback set 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 1 -- 172.21.15.37:0/4227138644 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8011d20 con 0x7f7bf0135220 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bee7fc700 10 monclient: dump: 2024-09-16T07:25:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.293 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 10 monclient: shutdown 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 -- 172.21.15.37:0/4227138644 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0135220 msgr2=0x7f7bf0135650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 --2- 172.21.15.37:0/4227138644 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0135220 0x7f7bf0135650 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7bf006b7c0 tx=0x7f7be8009b30 comp rx=0 tx=0).stop 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 -- 172.21.15.37:0/4227138644 shutdown_connections 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 --2- 172.21.15.37:0/4227138644 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0135220 0x7f7bf0135650 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 --2- 172.21.15.37:0/4227138644 >> v2:172.21.15.37:3300/0 conn(0x7f7bf006d8f0 0x7f7bf006b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 --2- 172.21.15.37:0/4227138644 >> v2:172.21.15.143:3300/0 conn(0x7f7bf006a750 0x7f7bf006ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 -- 172.21.15.37:0/4227138644 >> 172.21.15.37:0/4227138644 conn(0x7f7bf00a1530 msgr2=0x7f7bf00a1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 -- 172.21.15.37:0/4227138644 shutdown_connections 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.274+0000 7f7bf4c15700 1 -- 172.21.15.37:0/4227138644 wait complete. 2024-09-16T07:25:04.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: build_initial_monmap 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: monmap: 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 Processor -- start 2024-09-16T07:25:04.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 -- start start 2024-09-16T07:25:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: init 2024-09-16T07:25:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:25:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf006a750 0x7f7bf01ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient(hunting): picked mon.noname-c con 0x7f7bf006a750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf006d8f0 0x7f7bf01342c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient(hunting): picked mon.noname-a con 0x7f7bf006d8f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf0135220 0x7f7bf0134800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient(hunting): picked mon.noname-b con 0x7f7bf0135220 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf0125800 con 0x7f7bf006d8f0 2024-09-16T07:25:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf01caa00 con 0x7f7bf0135220 2024-09-16T07:25:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf0124750 con 0x7f7bf006a750 2024-09-16T07:25:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:04.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient(hunting): authenticate will time out at 1224.765322s 2024-09-16T07:25:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf006d8f0 0x7f7bf01342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf006d8f0 0x7f7bf01342c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39424/0 (socket says 172.21.15.37:39424) 2024-09-16T07:25:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 1 -- 172.21.15.37:0/1660989714 learned_addr learned my addr 172.21.15.37:0/1660989714 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 10 monclient(hunting): get_auth_request con 0x7f7bf006d8f0 auth_method 0 2024-09-16T07:25:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bef7fe700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf006a750 0x7f7bf01ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beffff700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf0135220 0x7f7bf0134800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bef7fe700 10 monclient(hunting): get_auth_request con 0x7f7bf006a750 auth_method 0 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beffff700 10 monclient(hunting): get_auth_request con 0x7f7bf0135220 auth_method 0 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 10 monclient(hunting): handle_auth_done global_id 54189 payload 824 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 1 -- 172.21.15.37:0/1660989714 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf006a750 msgr2=0x7f7bf01ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf006a750 0x7f7bf01ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 1 -- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf0135220 msgr2=0x7f7bf0134800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf0135220 0x7f7bf0134800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 10 monclient: _finish_hunting 0 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 1 monclient: found mon.noname-a 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 1 -- 172.21.15.37:0/1660989714 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be80094b0 con 0x7f7bf006d8f0 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beffff700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf0135220 0x7f7bf0134800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:25:04.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bef7fe700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf006a750 0x7f7bf01ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7beeffd700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf006d8f0 0x7f7bf01342c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800efc0 tx=0x7f7bd800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 1 -- 172.21.15.37:0/1660989714 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd800cbb0 con 0x7f7bf006d8f0 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 10 monclient: dump: 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 10 monclient: _finish_auth 0 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 10 monclient: _check_auth_tickets 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 5 monclient: authenticate success, global_id 54189 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: _renew_subs 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 1 -- 172.21.15.37:0/1660989714 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd800f040 con 0x7f7bf006d8f0 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 -- 172.21.15.37:0/1660989714 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf009e270 con 0x7f7bf006d8f0 2024-09-16T07:25:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 1 -- 172.21.15.37:0/1660989714 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8014480 con 0x7f7bf006d8f0 2024-09-16T07:25:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 10 monclient: dump: 2024-09-16T07:25:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bedffb700 4 set_mon_vals no callback set 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: _renew_subs 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 -- 172.21.15.37:0/1660989714 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf0072680 con 0x7f7bf006d8f0 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 1 -- 172.21.15.37:0/1660989714 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f7bd80145e0 con 0x7f7bf006d8f0 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:04.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7be7fff700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7bd0035600 0x7f7bd0037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.278+0000 7f7bf4c15700 1 -- 172.21.15.37:0/1660989714 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf0067890 con 0x7f7bf006d8f0 2024-09-16T07:25:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.282+0000 7f7be7fff700 1 -- 172.21.15.37:0/1660989714 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f7bd8048150 con 0x7f7bf006d8f0 2024-09-16T07:25:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.282+0000 7f7bef7fe700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7bd0035600 0x7f7bd0037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.282+0000 7f7bef7fe700 10 monclient: get_auth_request con 0x7f7bd0035600 auth_method 0 2024-09-16T07:25:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.282+0000 7f7bef7fe700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7bd0035600 0x7f7bd0037ac0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009a10 tx=0x7f7be0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.286+0000 7f7be7fff700 1 -- 172.21.15.37:0/1660989714 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7bd801d410 con 0x7f7bf006d8f0 2024-09-16T07:25:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.286+0000 7f7be7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.286+0000 7f7be7fff700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:04.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.454+0000 7f7bf4c15700 1 -- 172.21.15.37:0/1660989714 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bf0135b90 con 0x7f7bd0035600 2024-09-16T07:25:04.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.454+0000 7f7be7fff700 1 -- 172.21.15.37:0/1660989714 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7be00096a0 con 0x7f7bd0035600 2024-09-16T07:25:04.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 -- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7bd0035600 msgr2=0x7f7bd0037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7bd0035600 0x7f7bd0037ac0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009a10 tx=0x7f7be0008040 comp rx=0 tx=0).stop 2024-09-16T07:25:04.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 10 monclient: shutdown 2024-09-16T07:25:04.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:04.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 -- 172.21.15.37:0/1660989714 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf006d8f0 msgr2=0x7f7bf01342c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf006d8f0 0x7f7bf01342c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800efc0 tx=0x7f7bd800c3a0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 -- 172.21.15.37:0/1660989714 shutdown_connections 2024-09-16T07:25:04.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf0135220 0x7f7bf0134800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf006d8f0 0x7f7bf01342c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7bd0035600 0x7f7bd0037ac0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 --2- 172.21.15.37:0/1660989714 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf006a750 0x7f7bf01ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 -- 172.21.15.37:0/1660989714 >> 172.21.15.37:0/1660989714 conn(0x7f7bf00a1530 msgr2=0x7f7bf00a3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:04.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 -- 172.21.15.37:0/1660989714 shutdown_connections 2024-09-16T07:25:04.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.458+0000 7f7bf4c15700 1 -- 172.21.15.37:0/1660989714 wait complete. 2024-09-16T07:25:04.465 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T07:25:04.479 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T07:25:04.479 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T07:25:04.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.602+0000 7f7063793700 10 monclient: get_monmap_and_config 2024-09-16T07:25:04.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.602+0000 7f7063793700 10 monclient: build_initial_monmap 2024-09-16T07:25:04.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.602+0000 7f7063793700 10 monclient: monmap: 2024-09-16T07:25:04.605 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:04.604981+0000 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:04.604981+0000 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:04.606 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.602+0000 7f7063793700 1 Processor -- start 2024-09-16T07:25:04.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.602+0000 7f7063793700 1 -- start start 2024-09-16T07:25:04.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.602+0000 7f7063793700 10 monclient: init 2024-09-16T07:25:04.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f705c099e00 0x7f705c09a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient(hunting): picked mon.noname-a con 0x7f705c099e00 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f705c09d580 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f705c09a720 0x7f705c139d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient(hunting): picked mon.noname-c con 0x7f705c09a720 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705c09c670 con 0x7f705c09d580 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705c09b380 con 0x7f705c099e00 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705c09cba0 con 0x7f705c09a720 2024-09-16T07:25:04.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient(hunting): authenticate will time out at 1225.092419s 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f705c099e00 0x7f705c09a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7060d2e700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f705c09d580 0x7f705c136600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54450/0 (socket says 172.21.15.37:54450) 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f705c099e00 0x7f705c09a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39428/0 (socket says 172.21.15.37:39428) 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7060d2e700 1 -- 172.21.15.37:0/153319632 learned_addr learned my addr 172.21.15.37:0/153319632 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 10 monclient(hunting): get_auth_request con 0x7f705c099e00 auth_method 0 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7061d30700 1 --2- 172.21.15.37:0/153319632 >> v2:172.21.15.143:3300/0 conn(0x7f705c09a720 0x7f705c139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7061d30700 10 monclient(hunting): get_auth_request con 0x7f705c09a720 auth_method 0 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7061d30700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7061d30700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7061d30700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7061d30700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7061d30700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7061d30700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 10 monclient(hunting): handle_auth_done global_id 54192 payload 274 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 1 -- 172.21.15.37:0/153319632 >> v2:172.21.15.143:3300/0 conn(0x7f705c09a720 msgr2=0x7f705c139d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 1 --2- 172.21.15.37:0/153319632 >> v2:172.21.15.143:3300/0 conn(0x7f705c09a720 0x7f705c139d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 1 -- 172.21.15.37:0/153319632 >> v1:172.21.15.37:6789/0 conn(0x7f705c09d580 legacy=0x7f705c136600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 10 monclient: _finish_hunting 0 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 1 monclient: found mon.noname-a 2024-09-16T07:25:04.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 1 -- 172.21.15.37:0/153319632 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761338357 0 0) 0x7f705c09c670 con 0x7f705c09d580 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 1 -- 172.21.15.37:0/153319632 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c09c300 con 0x7f705c099e00 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f706152f700 1 --2- 172.21.15.37:0/153319632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f705c099e00 0x7f705c09a1e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f705c071de0 tx=0x7f705800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c3d129aaa4717c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 1 -- 172.21.15.37:0/153319632 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705800dc70 con 0x7f705c099e00 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 10 monclient: dump: 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.615 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 10 monclient: _finish_auth 0 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 10 monclient: _check_auth_tickets 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 1 -- 172.21.15.37:0/153319632 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705800ddd0 con 0x7f705c099e00 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 5 monclient: authenticate success, global_id 54192 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:04.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 1 -- 172.21.15.37:0/153319632 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7058010690 con 0x7f705c099e00 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 4 set_mon_vals no callback set 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7053fff700 10 monclient: dump: 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.617 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 10 monclient: shutdown 2024-09-16T07:25:04.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:04.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 -- 172.21.15.37:0/153319632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f705c099e00 msgr2=0x7f705c09a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 --2- 172.21.15.37:0/153319632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f705c099e00 0x7f705c09a1e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f705c071de0 tx=0x7f705800bdd0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 -- 172.21.15.37:0/153319632 shutdown_connections 2024-09-16T07:25:04.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 --2- 172.21.15.37:0/153319632 >> v2:172.21.15.143:3300/0 conn(0x7f705c09a720 0x7f705c139d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 --2- 172.21.15.37:0/153319632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f705c099e00 0x7f705c09a1e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 -- 172.21.15.37:0/153319632 >> 172.21.15.37:0/153319632 conn(0x7f705c0a4230 msgr2=0x7f705c0a4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:04.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 -- 172.21.15.37:0/153319632 shutdown_connections 2024-09-16T07:25:04.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.606+0000 7f7063793700 1 -- 172.21.15.37:0/153319632 wait complete. 2024-09-16T07:25:04.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient: build_initial_monmap 2024-09-16T07:25:04.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient: monmap: 2024-09-16T07:25:04.624 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:04.624 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:04.624 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:04.624 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:04.624 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:04.624 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:04.624 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 1 Processor -- start 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 1 -- start start 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient: init 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f705c099e00 0x7f705c1ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient(hunting): picked mon.noname-b con 0x7f705c099e00 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f705c09a720 0x7f705c1caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient(hunting): picked mon.noname-a con 0x7f705c09a720 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:04.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f705c09d580 0x7f705c1a6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient(hunting): picked mon.noname-c con 0x7f705c09d580 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c1aa330 con 0x7f705c09a720 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c1aa8f0 con 0x7f705c099e00 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705c1aabd0 con 0x7f705c09d580 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient(hunting): authenticate will time out at 1225.098402s 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f705c09d580 0x7f705c1a6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f705c09d580 0x7f705c1a6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35006/0 (socket says 172.21.15.37:35006) 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 1 -- 172.21.15.37:0/2487123902 learned_addr learned my addr 172.21.15.37:0/2487123902 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 10 monclient(hunting): get_auth_request con 0x7f705c09d580 auth_method 0 2024-09-16T07:25:04.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7060d2e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f705c09a720 0x7f705c1caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7060d2e700 10 monclient(hunting): get_auth_request con 0x7f705c09a720 auth_method 0 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7060d2e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7060d2e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7060d2e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f706152f700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f705c099e00 0x7f705c1ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f706152f700 10 monclient(hunting): get_auth_request con 0x7f705c099e00 auth_method 0 2024-09-16T07:25:04.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f706152f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f706152f700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f706152f700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7060d2e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7060d2e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7060d2e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f706152f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f706152f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f706152f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 10 monclient(hunting): handle_auth_done global_id 44579 payload 824 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 1 -- 172.21.15.37:0/2487123902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f705c099e00 msgr2=0x7f705c1ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f705c099e00 0x7f705c1ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 1 -- 172.21.15.37:0/2487123902 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f705c09a720 msgr2=0x7f705c1caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f705c09a720 0x7f705c1caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 10 monclient: _finish_hunting 0 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 1 monclient: found mon.noname-c 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 1 -- 172.21.15.37:0/2487123902 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705800b750 con 0x7f705c09d580 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7061d30700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f705c09d580 0x7f705c1a6c90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f705400efc0 tx=0x7f705400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 1 -- 172.21.15.37:0/2487123902 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054009ea0 con 0x7f705c09d580 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 10 monclient: dump: 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.629 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 10 monclient: _finish_auth 0 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 5 monclient: authenticate success, global_id 44579 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient: _renew_subs 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 1 -- 172.21.15.37:0/2487123902 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705400f040 con 0x7f705c09d580 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:04.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705c1a93d0 con 0x7f705c09d580 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 1 -- 172.21.15.37:0/2487123902 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705400ced0 con 0x7f705c09d580 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f7051ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7051ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.610+0000 7f70537fe700 4 set_mon_vals no callback set 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7051ffb700 10 monclient: dump: 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.631 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7063793700 10 monclient: _renew_subs 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7063793700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705c09b070 con 0x7f705c09d580 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7063793700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f705c1a88c0 con 0x7f705c09d580 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7051ffb700 1 -- 172.21.15.37:0/2487123902 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f7054014480 con 0x7f705c09d580 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7051ffb700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7040035600 0x7f7040037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7051ffb700 1 -- 172.21.15.37:0/2487123902 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f70540484e0 con 0x7f705c09d580 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7051ffb700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] conn(0x7f704003b2a0 0x7f704003d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7051ffb700 1 -- 172.21.15.37:0/2487123902 --> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f704003dd70 con 0x7f704003b2a0 2024-09-16T07:25:04.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7051ffb700 1 -- 172.21.15.37:0/2487123902 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7f705404ab70 con 0x7f705c09d580 2024-09-16T07:25:04.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7051ffb700 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T07:25:04.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f706152f700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7040035600 0x7f7040037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f706152f700 10 monclient: get_auth_request con 0x7f7040035600 auth_method 0 2024-09-16T07:25:04.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7060d2e700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] conn(0x7f704003b2a0 0x7f704003d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7060d2e700 10 monclient: get_auth_request con 0x7f704003b2a0 auth_method 0 2024-09-16T07:25:04.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f7060d2e700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] conn(0x7f704003b2a0 0x7f704003d6c0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.614+0000 7f706152f700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7040035600 0x7f7040037ac0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7058000c00 tx=0x7f7058008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.618+0000 7f7051ffb700 1 -- 172.21.15.37:0/2487123902 <== osd.2 v2:172.21.15.37:6816/265940208 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f704003dd70 con 0x7f704003b2a0 2024-09-16T07:25:04.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.634+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 --> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f705c09ab70 con 0x7f704003b2a0 2024-09-16T07:25:04.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.634+0000 7f7051ffb700 1 -- 172.21.15.37:0/2487123902 <== osd.2 v2:172.21.15.37:6816/265940208 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f704c004b50 con 0x7f704003b2a0 2024-09-16T07:25:04.639 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:25:04.639 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T07:25:04.639 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.634+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 >> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] conn(0x7f704003b2a0 msgr2=0x7f704003d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.634+0000 7f7063793700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.37:6816/265940208,v1:172.21.15.37:6817/265940208] conn(0x7f704003b2a0 0x7f704003d6c0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.634+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7040035600 msgr2=0x7f7040037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.634+0000 7f7063793700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f7040035600 0x7f7040037ac0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7058000c00 tx=0x7f7058008040 comp rx=0 tx=0).stop 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.638+0000 7f7063793700 10 monclient: shutdown 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.638+0000 7f7063793700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.638+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f705c09d580 msgr2=0x7f705c1a6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.638+0000 7f7063793700 1 --2- 172.21.15.37:0/2487123902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f705c09d580 0x7f705c1a6c90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f705400efc0 tx=0x7f705400c3a0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.638+0000 7f7061d30700 1 -- 172.21.15.37:0/2487123902 reap_dead start 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.638+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 shutdown_connections 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.638+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 >> 172.21.15.37:0/2487123902 conn(0x7f705c0a4230 msgr2=0x7f705c0a1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:04.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.638+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 shutdown_connections 2024-09-16T07:25:04.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.638+0000 7f7063793700 1 -- 172.21.15.37:0/2487123902 wait complete. 2024-09-16T07:25:04.652 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T07:25:04.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.766+0000 7f697d70f700 10 monclient: get_monmap_and_config 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.766+0000 7f697d70f700 10 monclient: build_initial_monmap 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.766+0000 7f697d70f700 10 monclient: monmap: 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:04.771006+0000 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:04.771006+0000 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:04.772 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:04.773 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:04.773 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 Processor -- start 2024-09-16T07:25:04.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 -- start start 2024-09-16T07:25:04.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient: init 2024-09-16T07:25:04.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:04.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f69780a0e90 0x7f69780a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient(hunting): picked mon.noname-a con 0x7f69780a0e90 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a1270 0x7f69780a2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient(hunting): picked mon.noname-b con 0x7f69780a1270 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f69780a2b80 0x7f69780a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient(hunting): picked mon.noname-c con 0x7f69780a2b80 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697806cf50 con 0x7f69780a0e90 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697809f810 con 0x7f69780a2b80 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697809f250 con 0x7f69780a1270 2024-09-16T07:25:04.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient(hunting): authenticate will time out at 1225.256867s 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a1270 0x7f69780a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f69780a0e90 0x7f69780a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f69780a0e90 0x7f69780a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39436/0 (socket says 172.21.15.37:39436) 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 1 -- 172.21.15.37:0/1558329589 learned_addr learned my addr 172.21.15.37:0/1558329589 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 10 monclient(hunting): get_auth_request con 0x7f69780a0e90 auth_method 0 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 10 monclient(hunting): get_auth_request con 0x7f69780a1270 auth_method 0 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697cf0e700 1 --2- 172.21.15.37:0/1558329589 >> v2:172.21.15.143:3300/0 conn(0x7f69780a2b80 0x7f69780a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697cf0e700 10 monclient(hunting): get_auth_request con 0x7f69780a2b80 auth_method 0 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697cf0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697cf0e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697cf0e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 10 monclient(hunting): handle_auth_done global_id 44582 payload 274 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 1 -- 172.21.15.37:0/1558329589 >> v2:172.21.15.143:3300/0 conn(0x7f69780a2b80 msgr2=0x7f69780a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:04.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 1 --2- 172.21.15.37:0/1558329589 >> v2:172.21.15.143:3300/0 conn(0x7f69780a2b80 0x7f69780a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 1 -- 172.21.15.37:0/1558329589 >> v2:172.21.15.37:3300/0 conn(0x7f69780a0e90 msgr2=0x7f69780a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 1 --2- 172.21.15.37:0/1558329589 >> v2:172.21.15.37:3300/0 conn(0x7f69780a0e90 0x7f69780a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 10 monclient: _finish_hunting 0 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 1 monclient: found mon.noname-b 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 1 -- 172.21.15.37:0/1558329589 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697809f530 con 0x7f69780a1270 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f6977fff700 1 --2- 172.21.15.37:0/1558329589 >> v2:172.21.15.37:3300/0 conn(0x7f69780a0e90 0x7f69780a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f696f7fe700 1 --2- 172.21.15.37:0/1558329589 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a1270 0x7f69780a2640 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f69780a20a0 tx=0x7f6960009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=208eeac60b73d3ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 1 -- 172.21.15.37:0/1558329589 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960011830 con 0x7f69780a1270 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:25:04.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 10 monclient: dump: 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 10 monclient: _finish_auth 0 2024-09-16T07:25:04.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 10 monclient: _check_auth_tickets 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 1 -- 172.21.15.37:0/1558329589 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6960011990 con 0x7f69780a1270 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 5 monclient: authenticate success, global_id 44582 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 1 -- 172.21.15.37:0/1558329589 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960011d20 con 0x7f69780a1270 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 4 set_mon_vals no callback set 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f69777fe700 10 monclient: dump: 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.782 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 10 monclient: shutdown 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 -- 172.21.15.37:0/1558329589 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a1270 msgr2=0x7f69780a2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 --2- 172.21.15.37:0/1558329589 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a1270 0x7f69780a2640 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f69780a20a0 tx=0x7f6960009b30 comp rx=0 tx=0).stop 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 -- 172.21.15.37:0/1558329589 shutdown_connections 2024-09-16T07:25:04.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 --2- 172.21.15.37:0/1558329589 >> v2:172.21.15.143:3300/0 conn(0x7f69780a2b80 0x7f69780a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 --2- 172.21.15.37:0/1558329589 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a1270 0x7f69780a2640 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 --2- 172.21.15.37:0/1558329589 >> v2:172.21.15.37:3300/0 conn(0x7f69780a0e90 0x7f69780a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 -- 172.21.15.37:0/1558329589 >> 172.21.15.37:0/1558329589 conn(0x7f69780a4740 msgr2=0x7f69780a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:04.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 -- 172.21.15.37:0/1558329589 shutdown_connections 2024-09-16T07:25:04.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.770+0000 7f697d70f700 1 -- 172.21.15.37:0/1558329589 wait complete. 2024-09-16T07:25:04.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient: build_initial_monmap 2024-09-16T07:25:04.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient: monmap: 2024-09-16T07:25:04.784 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 1 Processor -- start 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 1 -- start start 2024-09-16T07:25:04.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient: init 2024-09-16T07:25:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:25:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a0e90 0x7f69781c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient(hunting): picked mon.noname-c con 0x7f69780a0e90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69780a1270 0x7f69781c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient(hunting): picked mon.noname-b con 0x7f69780a1270 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69780a2b80 0x7f69781c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient(hunting): picked mon.noname-a con 0x7f69780a2b80 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69781aafd0 con 0x7f69780a2b80 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697809dfc0 con 0x7f69780a1270 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69781a8d20 con 0x7f69780a0e90 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient(hunting): authenticate will time out at 1225.261836s 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6977fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a0e90 0x7f69781c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6977fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a0e90 0x7f69781c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35022/0 (socket says 172.21.15.37:35022) 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6977fff700 1 -- 172.21.15.37:0/4107207782 learned_addr learned my addr 172.21.15.37:0/4107207782 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6977fff700 10 monclient(hunting): get_auth_request con 0x7f69780a0e90 auth_method 0 2024-09-16T07:25:04.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6977fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6977fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6977fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69780a2b80 0x7f69781c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6977fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6977fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6977fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f696f7fe700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69780a1270 0x7f69781c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 10 monclient(hunting): get_auth_request con 0x7f69780a2b80 auth_method 0 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f696f7fe700 10 monclient(hunting): get_auth_request con 0x7f69780a1270 auth_method 0 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f696f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:04.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f696f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f696f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 10 monclient(hunting): handle_auth_done global_id 54204 payload 824 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 1 -- 172.21.15.37:0/4107207782 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a0e90 msgr2=0x7f69781c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a0e90 0x7f69781c6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 1 -- 172.21.15.37:0/4107207782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69780a1270 msgr2=0x7f69781c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69780a1270 0x7f69781c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 10 monclient: _finish_hunting 0 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 1 monclient: found mon.noname-a 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 1 -- 172.21.15.37:0/4107207782 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6968009570 con 0x7f69780a2b80 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697cf0e700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69780a2b80 0x7f69781c9ec0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6970007d90 tx=0x7f697000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 1 -- 172.21.15.37:0/4107207782 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6970018070 con 0x7f69780a2b80 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 10 monclient: dump: 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.793 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 10 monclient: _finish_auth 0 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 10 monclient: _check_auth_tickets 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 5 monclient: authenticate success, global_id 54204 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient: _renew_subs 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 1 -- 172.21.15.37:0/4107207782 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69600094b0 con 0x7f69780a2b80 2024-09-16T07:25:04.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 1 -- 172.21.15.37:0/4107207782 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697000ead0 con 0x7f69780a2b80 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 1 -- 172.21.15.37:0/4107207782 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697000ee60 con 0x7f69780a2b80 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient: _renew_subs 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f6976ffd700 4 set_mon_vals no callback set 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f697d70f700 1 -- 172.21.15.37:0/4107207782 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697806d780 con 0x7f69780a2b80 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.774+0000 7f69757fa700 10 monclient: dump: 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.795 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.778+0000 7f697d70f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.778+0000 7f697d70f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.778+0000 7f697d70f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.778+0000 7f697d70f700 1 -- 172.21.15.37:0/4107207782 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69780a2110 con 0x7f69780a2b80 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.778+0000 7f69757fa700 1 -- 172.21.15.37:0/4107207782 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f6970014420 con 0x7f69780a2b80 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.778+0000 7f69757fa700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6958035600 0x7f6958037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.778+0000 7f69757fa700 1 -- 172.21.15.37:0/4107207782 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f6970047770 con 0x7f69780a2b80 2024-09-16T07:25:04.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.778+0000 7f6977fff700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6958035600 0x7f6958037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:04.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.778+0000 7f6977fff700 10 monclient: get_auth_request con 0x7f6958035600 auth_method 0 2024-09-16T07:25:04.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.778+0000 7f6977fff700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6958035600 0x7f6958037ac0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f69781a9010 tx=0x7f69680092e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:04.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.786+0000 7f69757fa700 1 -- 172.21.15.37:0/4107207782 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f69700126a0 con 0x7f69780a2b80 2024-09-16T07:25:04.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.786+0000 7f69757fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:04.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.786+0000 7f69757fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:04.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.954+0000 7f697d70f700 1 -- 172.21.15.37:0/4107207782 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f69780a16a0 con 0x7f6958035600 2024-09-16T07:25:04.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.954+0000 7f69757fa700 1 -- 172.21.15.37:0/4107207782 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6968011b40 con 0x7f6958035600 2024-09-16T07:25:04.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 -- 172.21.15.37:0/4107207782 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6958035600 msgr2=0x7f6958037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6958035600 0x7f6958037ac0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f69781a9010 tx=0x7f69680092e0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 10 monclient: shutdown 2024-09-16T07:25:04.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:04.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 -- 172.21.15.37:0/4107207782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69780a2b80 msgr2=0x7f69781c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:04.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69780a2b80 0x7f69781c9ec0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6970007d90 tx=0x7f697000e3c0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 -- 172.21.15.37:0/4107207782 shutdown_connections 2024-09-16T07:25:04.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69780a2b80 0x7f69781c9ec0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69780a1270 0x7f69781c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f6958035600 0x7f6958037ac0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 --2- 172.21.15.37:0/4107207782 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69780a0e90 0x7f69781c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:04.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 -- 172.21.15.37:0/4107207782 >> 172.21.15.37:0/4107207782 conn(0x7f69780a4740 msgr2=0x7f69780a70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:04.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 -- 172.21.15.37:0/4107207782 shutdown_connections 2024-09-16T07:25:04.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:04.958+0000 7f697d70f700 1 -- 172.21.15.37:0/4107207782 wait complete. 2024-09-16T07:25:04.962 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T07:25:04.978 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T07:25:04.978 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T07:25:05.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient: get_monmap_and_config 2024-09-16T07:25:05.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient: build_initial_monmap 2024-09-16T07:25:05.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient: monmap: 2024-09-16T07:25:05.101 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:05.101 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.102 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:05.100752+0000 2024-09-16T07:25:05.102 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:05.100752+0000 2024-09-16T07:25:05.102 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:05.102 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.102 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:05.102 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:05.102 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:05.102 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:05.102 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:05.102 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 1 Processor -- start 2024-09-16T07:25:05.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 1 -- start start 2024-09-16T07:25:05.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient: init 2024-09-16T07:25:05.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:05.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409f3c0 0x7f2d1409f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient(hunting): picked mon.noname-b con 0x7f2d1409f3c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2d1409cd00 0x7f2d1409fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient(hunting): picked mon.noname-a con 0x7f2d1409cd00 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d14133190 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d140a0220 con 0x7f2d14133190 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1406c590 con 0x7f2d1409cd00 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d14099be0 con 0x7f2d1409f3c0 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19a84700 10 monclient(hunting): authenticate will time out at 1225.586893s 2024-09-16T07:25:05.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d18a82700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409f3c0 0x7f2d1409f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2d1409cd00 0x7f2d1409fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19283700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2d14133190 0x7f2d14133640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54466/0 (socket says 172.21.15.37:54466) 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2d1409cd00 0x7f2d1409fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39452/0 (socket says 172.21.15.37:39452) 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d18a82700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409f3c0 0x7f2d1409f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35036/0 (socket says 172.21.15.37:35036) 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d19283700 1 -- 172.21.15.37:0/1298551196 learned_addr learned my addr 172.21.15.37:0/1298551196 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 10 monclient(hunting): get_auth_request con 0x7f2d1409cd00 auth_method 0 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d18a82700 10 monclient(hunting): get_auth_request con 0x7f2d1409f3c0 auth_method 0 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d18a82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d18a82700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d18a82700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d18a82700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d18a82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d18a82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 10 monclient(hunting): handle_auth_done global_id 54207 payload 274 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 1 -- 172.21.15.37:0/1298551196 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409f3c0 msgr2=0x7f2d1409f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 1 --2- 172.21.15.37:0/1298551196 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409f3c0 0x7f2d1409f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 1 -- 172.21.15.37:0/1298551196 >> v1:172.21.15.37:6789/0 conn(0x7f2d14133190 legacy=0x7f2d14133640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 10 monclient: _finish_hunting 0 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 1 monclient: found mon.noname-a 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:05.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0bfff700 1 -- 172.21.15.37:0/1298551196 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1409d530 con 0x7f2d1409cd00 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d18a82700 1 --2- 172.21.15.37:0/1298551196 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409f3c0 0x7f2d1409f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0b7fe700 1 -- 172.21.15.37:0/1298551196 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765343502 0 0) 0x7f2d140a0220 con 0x7f2d14133190 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.098+0000 7f2d0b7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0bfff700 1 --2- 172.21.15.37:0/1298551196 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d1409cd00 0x7f2d1409fce0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2d10009850 tx=0x7f2d10009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3fbc1b3db0391e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 1 -- 172.21.15.37:0/1298551196 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d10011830 con 0x7f2d1409cd00 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 10 monclient: dump: 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.110 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 10 monclient: _finish_auth 0 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 5 monclient: authenticate success, global_id 54207 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 1 -- 172.21.15.37:0/1298551196 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d10011990 con 0x7f2d1409cd00 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:05.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 1 -- 172.21.15.37:0/1298551196 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d10011d40 con 0x7f2d1409cd00 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 4 set_mon_vals no callback set 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d0b7fe700 10 monclient: dump: 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.112 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 10 monclient: shutdown 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 1 -- 172.21.15.37:0/1298551196 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d1409cd00 msgr2=0x7f2d1409fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 1 --2- 172.21.15.37:0/1298551196 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d1409cd00 0x7f2d1409fce0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2d10009850 tx=0x7f2d10009b60 comp rx=0 tx=0).stop 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 1 -- 172.21.15.37:0/1298551196 shutdown_connections 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 1 --2- 172.21.15.37:0/1298551196 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d1409cd00 0x7f2d1409fce0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 1 --2- 172.21.15.37:0/1298551196 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409f3c0 0x7f2d1409f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 1 -- 172.21.15.37:0/1298551196 >> 172.21.15.37:0/1298551196 conn(0x7f2d140a04b0 msgr2=0x7f2d140a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 1 -- 172.21.15.37:0/1298551196 shutdown_connections 2024-09-16T07:25:05.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 1 -- 172.21.15.37:0/1298551196 wait complete. 2024-09-16T07:25:05.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 10 monclient: build_initial_monmap 2024-09-16T07:25:05.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.102+0000 7f2d19a84700 10 monclient: monmap: 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 Processor -- start 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 -- start start 2024-09-16T07:25:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient: init 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409cd00 0x7f2d141c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient(hunting): picked mon.noname-c con 0x7f2d1409cd00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d1409f3c0 0x7f2d14132590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient(hunting): picked mon.noname-a con 0x7f2d1409f3c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d14133190 0x7f2d14132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient(hunting): picked mon.noname-b con 0x7f2d14133190 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d141c7f10 con 0x7f2d1409f3c0 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d14123ab0 con 0x7f2d14133190 2024-09-16T07:25:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d140725f0 con 0x7f2d1409cd00 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient(hunting): authenticate will time out at 1225.593594s 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d1409f3c0 0x7f2d14132590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d18a82700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409cd00 0x7f2d141c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d1409f3c0 0x7f2d14132590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39468/0 (socket says 172.21.15.37:39468) 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d18a82700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409cd00 0x7f2d141c79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35040/0 (socket says 172.21.15.37:35040) 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 1 -- 172.21.15.37:0/570487231 learned_addr learned my addr 172.21.15.37:0/570487231 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 10 monclient(hunting): get_auth_request con 0x7f2d1409f3c0 auth_method 0 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d18a82700 10 monclient(hunting): get_auth_request con 0x7f2d1409cd00 auth_method 0 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d18a82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d18a82700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d18a82700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d18a82700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d18a82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d18a82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19283700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d14133190 0x7f2d14132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19283700 10 monclient(hunting): get_auth_request con 0x7f2d14133190 auth_method 0 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19283700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19283700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19283700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 10 monclient(hunting): handle_auth_done global_id 54213 payload 824 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 1 -- 172.21.15.37:0/570487231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409cd00 msgr2=0x7f2d141c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d1409cd00 0x7f2d141c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 1 -- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d14133190 msgr2=0x7f2d14132ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d14133190 0x7f2d14132ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 10 monclient: _finish_hunting 0 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 1 monclient: found mon.noname-a 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 1 -- 172.21.15.37:0/570487231 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d040094b0 con 0x7f2d1409f3c0 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0bfff700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d1409f3c0 0x7f2d14132590 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2d10000c00 tx=0x7f2d10004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 1 -- 172.21.15.37:0/570487231 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1001aa10 con 0x7f2d1409f3c0 2024-09-16T07:25:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 10 monclient: dump: 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 10 monclient: _finish_auth 0 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 10 monclient: _check_auth_tickets 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 5 monclient: authenticate success, global_id 54213 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 1 -- 172.21.15.37:0/570487231 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1001f070 con 0x7f2d1409f3c0 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient: _renew_subs 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d100094b0 con 0x7f2d1409f3c0 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 1 -- 172.21.15.37:0/570487231 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1001ad90 con 0x7f2d1409f3c0 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d097fa700 10 monclient: dump: 2024-09-16T07:25:05.125 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d0affd700 4 set_mon_vals no callback set 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient: _renew_subs 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d140703d0 con 0x7f2d1409f3c0 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.106+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2d14099e90 con 0x7f2d1409f3c0 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d097fa700 1 -- 172.21.15.37:0/570487231 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f2d1001b430 con 0x7f2d1409f3c0 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d097fa700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2d00035600 0x7f2d00037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d097fa700 1 -- 172.21.15.37:0/570487231 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f2d10021070 con 0x7f2d1409f3c0 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d097fa700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] conn(0x7f2d0003b410 0x7f2d0003d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d097fa700 1 -- 172.21.15.37:0/570487231 --> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d0003dee0 con 0x7f2d0003b410 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d097fa700 1 -- 172.21.15.37:0/570487231 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d1004b270 con 0x7f2d1409f3c0 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d097fa700 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d18a82700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2d00035600 0x7f2d00037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d18a82700 10 monclient: get_auth_request con 0x7f2d00035600 auth_method 0 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d19283700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] conn(0x7f2d0003b410 0x7f2d0003d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d19283700 10 monclient: get_auth_request con 0x7f2d0003b410 auth_method 0 2024-09-16T07:25:05.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d18a82700 1 -- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2d00035600 msgr2=0x7f2d00037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T07:25:05.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d18a82700 1 -- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2d00035600 msgr2=0x7f2d00037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T07:25:05.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d18a82700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2d00035600 0x7f2d00037ac0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f2d04009d80 tx=0x7f2d04009240 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T07:25:05.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d18a82700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2d00035600 0x7f2d00037ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T07:25:05.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d19283700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] conn(0x7f2d0003b410 0x7f2d0003d830 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.110+0000 7f2d097fa700 1 -- 172.21.15.37:0/570487231 <== osd.3 v2:172.21.15.143:6809/2588720019 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f2d0003dee0 con 0x7f2d0003b410 2024-09-16T07:25:05.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 --> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2d140a34b0 con 0x7f2d0003b410 2024-09-16T07:25:05.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d097fa700 1 -- 172.21.15.37:0/570487231 <== osd.3 v2:172.21.15.143:6809/2588720019 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2d140a34b0 con 0x7f2d0003b410 2024-09-16T07:25:05.133 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:25:05.133 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T07:25:05.133 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:25:05.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] conn(0x7f2d0003b410 msgr2=0x7f2d0003d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6809/2588720019,v1:172.21.15.143:6811/2588720019] conn(0x7f2d0003b410 0x7f2d0003d830 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2d00035600 msgr2=0x7f2d00037ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T07:25:05.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f2d00035600 0x7f2d00037ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 10 monclient: shutdown 2024-09-16T07:25:05.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:05.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d1409f3c0 msgr2=0x7f2d14132590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 --2- 172.21.15.37:0/570487231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d1409f3c0 0x7f2d14132590 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2d10000c00 tx=0x7f2d10004770 comp rx=0 tx=0).stop 2024-09-16T07:25:05.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19283700 1 -- 172.21.15.37:0/570487231 reap_dead start 2024-09-16T07:25:05.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 shutdown_connections 2024-09-16T07:25:05.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 >> 172.21.15.37:0/570487231 conn(0x7f2d140a04b0 msgr2=0x7f2d1406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:05.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 shutdown_connections 2024-09-16T07:25:05.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.130+0000 7f2d19a84700 1 -- 172.21.15.37:0/570487231 wait complete. 2024-09-16T07:25:05.146 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient: get_monmap_and_config 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient: build_initial_monmap 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient: monmap: 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:05.266060+0000 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:05.266060+0000 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:05.267 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:05.268 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 1 Processor -- start 2024-09-16T07:25:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 1 -- start start 2024-09-16T07:25:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient: init 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f302c0a0e90 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a04d0 0x7f302c0a1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient(hunting): picked mon.noname-b con 0x7f302c0a04d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f302c0a2d50 0x7f302c0a31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient(hunting): picked mon.noname-c con 0x7f302c0a2d50 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0a0900 con 0x7f302c0a0e90 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302c0a17b0 con 0x7f302c0a2d50 2024-09-16T07:25:05.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c06cf50 con 0x7f302c0a04d0 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.262+0000 7f303296f700 10 monclient(hunting): authenticate will time out at 1225.751920s 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303196d700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f302c0a0e90 0x7f302c0a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54482/0 (socket says 172.21.15.37:54482) 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303196d700 1 -- 172.21.15.37:0/1500873967 learned_addr learned my addr 172.21.15.37:0/1500873967 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 1 --2- 172.21.15.37:0/1500873967 >> v2:172.21.15.143:3300/0 conn(0x7f302c0a2d50 0x7f302c0a31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 10 monclient(hunting): get_auth_request con 0x7f302c0a2d50 auth_method 0 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303116c700 1 --2- 172.21.15.37:0/1500873967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a04d0 0x7f302c0a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303116c700 10 monclient(hunting): get_auth_request con 0x7f302c0a04d0 auth_method 0 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303116c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303116c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303116c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 1 -- 172.21.15.37:0/1500873967 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186877011 0 0) 0x7f302c0a0900 con 0x7f302c0a0e90 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient(hunting): _negotiate 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient(hunting): my global_id is 54216 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 1 -- 172.21.15.37:0/1500873967 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30180035d0 con 0x7f302c0a0e90 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303116c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303116c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303116c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 10 monclient(hunting): handle_auth_done global_id 6376 payload 274 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 1 -- 172.21.15.37:0/1500873967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a04d0 msgr2=0x7f302c0a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 1 --2- 172.21.15.37:0/1500873967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a04d0 0x7f302c0a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 1 -- 172.21.15.37:0/1500873967 >> v1:172.21.15.37:6789/0 conn(0x7f302c0a0e90 legacy=0x7f302c0a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 10 monclient: _finish_hunting 0 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 1 monclient: found mon.noname-c 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 1 -- 172.21.15.37:0/1500873967 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c0a36e0 con 0x7f302c0a2d50 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303216e700 1 --2- 172.21.15.37:0/1500873967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c0a2d50 0x7f302c0a31a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3020009850 tx=0x7f3020009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9899b9284631a748 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 1 -- 172.21.15.37:0/1500873967 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3020011830 con 0x7f302c0a2d50 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient: dump: 2024-09-16T07:25:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient: _finish_auth 0 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient: _check_auth_tickets 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 5 monclient: authenticate success, global_id 6376 2024-09-16T07:25:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 1 -- 172.21.15.37:0/1500873967 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3020011990 con 0x7f302c0a2d50 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 1 -- 172.21.15.37:0/1500873967 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3020011d00 con 0x7f302c0a2d50 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 4 set_mon_vals no callback set 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303096b700 10 monclient: dump: 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.277 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 10 monclient: shutdown 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 1 -- 172.21.15.37:0/1500873967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c0a2d50 msgr2=0x7f302c0a31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 1 --2- 172.21.15.37:0/1500873967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c0a2d50 0x7f302c0a31a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3020009850 tx=0x7f3020009b60 comp rx=0 tx=0).stop 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 1 -- 172.21.15.37:0/1500873967 shutdown_connections 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 1 --2- 172.21.15.37:0/1500873967 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c0a2d50 0x7f302c0a31a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 1 --2- 172.21.15.37:0/1500873967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a04d0 0x7f302c0a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 1 -- 172.21.15.37:0/1500873967 >> 172.21.15.37:0/1500873967 conn(0x7f302c0a4740 msgr2=0x7f302c0a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:05.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 1 -- 172.21.15.37:0/1500873967 shutdown_connections 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 1 -- 172.21.15.37:0/1500873967 wait complete. 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 10 monclient: build_initial_monmap 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.266+0000 7f303296f700 10 monclient: monmap: 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 Processor -- start 2024-09-16T07:25:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 -- start start 2024-09-16T07:25:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient: init 2024-09-16T07:25:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:25:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f302c0a04d0 0x7f302c13b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient(hunting): picked mon.noname-a con 0x7f302c0a04d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c0a0e90 0x7f302c13bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient(hunting): picked mon.noname-b con 0x7f302c0a0e90 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a2d50 0x7f302c13c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient(hunting): picked mon.noname-c con 0x7f302c0a2d50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c1247e0 con 0x7f302c0a04d0 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c09dcf0 con 0x7f302c0a0e90 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c1253b0 con 0x7f302c0a2d50 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient(hunting): authenticate will time out at 1225.756929s 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303196d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f302c0a04d0 0x7f302c13b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a2d50 0x7f302c13c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a2d50 0x7f302c13c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35060/0 (socket says 172.21.15.37:35060) 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303196d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f302c0a04d0 0x7f302c13b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39476/0 (socket says 172.21.15.37:39476) 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 1 -- 172.21.15.37:0/1783910714 learned_addr learned my addr 172.21.15.37:0/1783910714 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 10 monclient(hunting): get_auth_request con 0x7f302c0a2d50 auth_method 0 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303196d700 10 monclient(hunting): get_auth_request con 0x7f302c0a04d0 auth_method 0 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303196d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303196d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303196d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303196d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303196d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303196d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303116c700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c0a0e90 0x7f302c13bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303116c700 10 monclient(hunting): get_auth_request con 0x7f302c0a0e90 auth_method 0 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303116c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303116c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303116c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 10 monclient(hunting): handle_auth_done global_id 44597 payload 824 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 1 -- 172.21.15.37:0/1783910714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c0a0e90 msgr2=0x7f302c13bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c0a0e90 0x7f302c13bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 1 -- 172.21.15.37:0/1783910714 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f302c0a04d0 msgr2=0x7f302c13b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f302c0a04d0 0x7f302c13b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 10 monclient: _finish_hunting 0 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 1 monclient: found mon.noname-c 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 1 -- 172.21.15.37:0/1783910714 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30200094b0 con 0x7f302c0a2d50 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303216e700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a2d50 0x7f302c13c270 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f30200090b0 tx=0x7f3020004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 1 -- 172.21.15.37:0/1783910714 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30200044d0 con 0x7f302c0a2d50 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 10 monclient: dump: 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 10 monclient: _finish_auth 0 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 5 monclient: authenticate success, global_id 44597 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient: _renew_subs 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:05.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 -- 172.21.15.37:0/1783910714 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c06bd00 con 0x7f302c0a2d50 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 1 -- 172.21.15.37:0/1783910714 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302001f070 con 0x7f302c0a2d50 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient: _renew_subs 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 -- 172.21.15.37:0/1783910714 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c126860 con 0x7f302c0a2d50 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 1 -- 172.21.15.37:0/1783910714 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30200038c0 con 0x7f302c0a2d50 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 10 monclient: dump: 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301f7fe700 4 set_mon_vals no callback set 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 1 -- 172.21.15.37:0/1783910714 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f3020003a20 con 0x7f302c0a2d50 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f303296f700 1 -- 172.21.15.37:0/1783910714 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3014005100 con 0x7f302c0a2d50 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3010035600 0x7f3010037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.270+0000 7f301e7fc700 1 -- 172.21.15.37:0/1783910714 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f3020021070 con 0x7f302c0a2d50 2024-09-16T07:25:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.274+0000 7f303196d700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3010035600 0x7f3010037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.274+0000 7f303196d700 10 monclient: get_auth_request con 0x7f3010035600 auth_method 0 2024-09-16T07:25:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.274+0000 7f303196d700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3010035600 0x7f3010037ac0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3024001560 tx=0x7f3024001a90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.278+0000 7f301e7fc700 1 -- 172.21.15.37:0/1783910714 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3020024380 con 0x7f302c0a2d50 2024-09-16T07:25:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.278+0000 7f301e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.278+0000 7f301e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:05.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.442+0000 7f303296f700 1 -- 172.21.15.37:0/1783910714 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3014000bc0 con 0x7f3010035600 2024-09-16T07:25:05.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f301e7fc700 1 -- 172.21.15.37:0/1783910714 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3014000bc0 con 0x7f3010035600 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 -- 172.21.15.37:0/1783910714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3010035600 msgr2=0x7f3010037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3010035600 0x7f3010037ac0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3024001560 tx=0x7f3024001a90 comp rx=0 tx=0).stop 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 10 monclient: shutdown 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 -- 172.21.15.37:0/1783910714 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a2d50 msgr2=0x7f302c13c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a2d50 0x7f302c13c270 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f30200090b0 tx=0x7f3020004770 comp rx=0 tx=0).stop 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 -- 172.21.15.37:0/1783910714 shutdown_connections 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f3010035600 0x7f3010037ac0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f302c0a2d50 0x7f302c13c270 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c0a0e90 0x7f302c13bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 --2- 172.21.15.37:0/1783910714 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f302c0a04d0 0x7f302c13b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 -- 172.21.15.37:0/1783910714 >> 172.21.15.37:0/1783910714 conn(0x7f302c0a4740 msgr2=0x7f302c09c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:05.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 -- 172.21.15.37:0/1783910714 shutdown_connections 2024-09-16T07:25:05.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.446+0000 7f303296f700 1 -- 172.21.15.37:0/1783910714 wait complete. 2024-09-16T07:25:05.452 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T07:25:05.466 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T07:25:05.466 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient: get_monmap_and_config 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient: build_initial_monmap 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient: monmap: 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:05.585192+0000 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:05.585192+0000 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:05.586 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:05.587 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:05.587 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:05.587 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:05.587 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 1 Processor -- start 2024-09-16T07:25:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 1 -- start start 2024-09-16T07:25:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient: init 2024-09-16T07:25:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T07:25:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3ec0295a0 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa3ec0248c0 0x7fa3ec0c0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fa3ec0248c0 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3ec0c1040 addr v1:172.21.15.143:6789/0 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec026820 con 0x7fa3ec0295a0 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ec027770 con 0x7fa3ec0c1040 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ec027a50 con 0x7fa3ec0248c0 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f3fff700 10 monclient(hunting): authenticate will time out at 1226.071247s 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f2ffd700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa3ec0295a0 0x7fa3ec023ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54498/0 (socket says 172.21.15.37:54498) 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f2ffd700 1 -- 172.21.15.37:0/578299034 learned_addr learned my addr 172.21.15.37:0/578299034 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f27fc700 1 --2- 172.21.15.37:0/578299034 >> v2:172.21.15.143:3300/0 conn(0x7fa3ec0248c0 0x7fa3ec0c0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f27fc700 10 monclient(hunting): get_auth_request con 0x7fa3ec0248c0 auth_method 0 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.582+0000 7fa3f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 1 -- 172.21.15.37:0/578299034 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717351960 0 0) 0x7fa3ec027770 con 0x7fa3ec0c1040 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient(hunting): _negotiate 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient(hunting): my global_id is 6385 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 1 -- 172.21.15.37:0/578299034 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc0035d0 con 0x7fa3ec0c1040 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 1 -- 172.21.15.37:0/578299034 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224428332 0 0) 0x7fa3ec026820 con 0x7fa3ec0295a0 2024-09-16T07:25:05.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient(hunting): _negotiate 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient(hunting): my global_id is 54222 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 1 -- 172.21.15.37:0/578299034 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ec027770 con 0x7fa3ec0295a0 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f27fc700 10 monclient(hunting): handle_auth_done global_id 6382 payload 274 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f27fc700 1 -- 172.21.15.37:0/578299034 >> v1:172.21.15.143:6789/0 conn(0x7fa3ec0c1040 legacy=0x7fa3ec0c3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f27fc700 1 -- 172.21.15.37:0/578299034 >> v1:172.21.15.37:6789/0 conn(0x7fa3ec0295a0 legacy=0x7fa3ec023ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f27fc700 10 monclient: _finish_hunting 0 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f27fc700 1 monclient: found mon.noname-c 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f27fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.143:3300/0 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f27fc700 1 -- 172.21.15.37:0/578299034 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec0bc670 con 0x7fa3ec0248c0 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f27fc700 1 --2- 172.21.15.37:0/578299034 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3ec0248c0 0x7fa3ec0c0b00 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009850 tx=0x7fa3e4009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82c852a95036b18b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 1 -- 172.21.15.37:0/578299034 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4011830 con 0x7fa3ec0248c0 2024-09-16T07:25:05.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:05.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient: dump: 2024-09-16T07:25:05.594 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.594 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.594 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.594 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.594 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.594 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.594 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient: _finish_auth 0 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 5 monclient: authenticate success, global_id 6382 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 1 -- 172.21.15.37:0/578299034 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e4011990 con 0x7fa3ec0248c0 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:05.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 4 set_mon_vals no callback set 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 1 -- 172.21.15.37:0/578299034 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4011d00 con 0x7fa3ec0248c0 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f1ffb700 10 monclient: dump: 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.596 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 10 monclient: shutdown 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 1 -- 172.21.15.37:0/578299034 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3ec0248c0 msgr2=0x7fa3ec0c0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 1 --2- 172.21.15.37:0/578299034 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3ec0248c0 0x7fa3ec0c0b00 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009850 tx=0x7fa3e4009b60 comp rx=0 tx=0).stop 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 1 -- 172.21.15.37:0/578299034 shutdown_connections 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 1 --2- 172.21.15.37:0/578299034 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3ec0248c0 0x7fa3ec0c0b00 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 1 -- 172.21.15.37:0/578299034 >> 172.21.15.37:0/578299034 conn(0x7fa3ec006440 msgr2=0x7fa3ec006850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 1 -- 172.21.15.37:0/578299034 shutdown_connections 2024-09-16T07:25:05.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 1 -- 172.21.15.37:0/578299034 wait complete. 2024-09-16T07:25:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 10 monclient: build_initial_monmap 2024-09-16T07:25:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 10 monclient: monmap: 2024-09-16T07:25:05.602 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 1 Processor -- start 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 1 -- start start 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.586+0000 7fa3f3fff700 10 monclient: init 2024-09-16T07:25:05.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3ec0295a0 0x7fa3ec1504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fa3ec0295a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa3ec0c1040 0x7fa3ec0b96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fa3ec0c1040 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3ec0b9c30 0x7fa3ec01f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient(hunting): picked mon.noname-b con 0x7fa3ec0b9c30 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec150a00 con 0x7fa3ec0295a0 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec024cb0 con 0x7fa3ec0b9c30 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ec151430 con 0x7fa3ec0c1040 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient(hunting): authenticate will time out at 1226.077015s 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3ec0295a0 0x7fa3ec1504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3ec0295a0 0x7fa3ec1504c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39490/0 (socket says 172.21.15.37:39490) 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 1 -- 172.21.15.37:0/713333870 learned_addr learned my addr 172.21.15.37:0/713333870 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 10 monclient(hunting): get_auth_request con 0x7fa3ec0295a0 auth_method 0 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f27fc700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa3ec0c1040 0x7fa3ec0b96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f27fc700 10 monclient(hunting): get_auth_request con 0x7fa3ec0c1040 auth_method 0 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f37fe700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3ec0b9c30 0x7fa3ec01f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f37fe700 10 monclient(hunting): get_auth_request con 0x7fa3ec0b9c30 auth_method 0 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 10 monclient(hunting): handle_auth_done global_id 54225 payload 824 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 1 -- 172.21.15.37:0/713333870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa3ec0c1040 msgr2=0x7fa3ec0b96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa3ec0c1040 0x7fa3ec0b96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 1 -- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3ec0b9c30 msgr2=0x7fa3ec01f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3ec0b9c30 0x7fa3ec01f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 10 monclient: _finish_hunting 0 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 1 monclient: found mon.noname-a 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 1 -- 172.21.15.37:0/713333870 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e40094b0 con 0x7fa3ec0295a0 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f2ffd700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3ec0295a0 0x7fa3ec1504c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800c4f0 tx=0x7fa3e800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 1 -- 172.21.15.37:0/713333870 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e800cea0 con 0x7fa3ec0295a0 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 10 monclient: dump: 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.608 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 10 monclient: _finish_auth 0 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 5 monclient: authenticate success, global_id 54225 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 1 -- 172.21.15.37:0/713333870 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e8005d90 con 0x7fa3ec0295a0 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient: _renew_subs 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ec023230 con 0x7fa3ec0295a0 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 1 -- 172.21.15.37:0/713333870 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e80072d0 con 0x7fa3ec0295a0 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:05.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 10 monclient: dump: 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f17fa700 4 set_mon_vals no callback set 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient: _renew_subs 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:05.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ec0a5dc0 con 0x7fa3ec0295a0 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3db7fe700 1 -- 172.21.15.37:0/713333870 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fa3e8007430 con 0x7fa3ec0295a0 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.590+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa3ec0bbcc0 con 0x7fa3ec0295a0 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3db7fe700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa3e0035600 0x7fa3e0037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3db7fe700 1 -- 172.21.15.37:0/713333870 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fa3e8046d80 con 0x7fa3ec0295a0 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3db7fe700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] conn(0x7fa3e003b410 0x7fa3e003d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3db7fe700 1 -- 172.21.15.37:0/713333870 --> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3e003dee0 con 0x7fa3e003b410 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3db7fe700 1 -- 172.21.15.37:0/713333870 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3e8049410 con 0x7fa3ec0295a0 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3db7fe700 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3f27fc700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa3e0035600 0x7fa3e0037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3f27fc700 10 monclient: get_auth_request con 0x7fa3e0035600 auth_method 0 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3f37fe700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] conn(0x7fa3e003b410 0x7fa3e003d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3f37fe700 10 monclient: get_auth_request con 0x7fa3e003b410 auth_method 0 2024-09-16T07:25:05.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3f27fc700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa3e0035600 0x7fa3e0037ac0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009850 tx=0x7fa3e4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3f37fe700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] conn(0x7fa3e003b410 0x7fa3e003d830 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.594+0000 7fa3db7fe700 1 -- 172.21.15.37:0/713333870 <== osd.4 v2:172.21.15.143:6808/3355421705 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fa3e003dee0 con 0x7fa3e003b410 2024-09-16T07:25:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.614+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 --> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa3ec008550 con 0x7fa3e003b410 2024-09-16T07:25:05.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.614+0000 7fa3db7fe700 1 -- 172.21.15.37:0/713333870 <== osd.4 v2:172.21.15.143:6808/3355421705 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa3ec008550 con 0x7fa3e003b410 2024-09-16T07:25:05.619 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:25:05.620 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T07:25:05.620 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:25:05.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] conn(0x7fa3e003b410 msgr2=0x7fa3e003d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:6808/3355421705,v1:172.21.15.143:6810/3355421705] conn(0x7fa3e003b410 0x7fa3e003d830 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa3e0035600 msgr2=0x7fa3e0037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa3e0035600 0x7fa3e0037ac0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009850 tx=0x7fa3e4008030 comp rx=0 tx=0).stop 2024-09-16T07:25:05.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 10 monclient: shutdown 2024-09-16T07:25:05.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:05.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3ec0295a0 msgr2=0x7fa3ec1504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 1 --2- 172.21.15.37:0/713333870 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3ec0295a0 0x7fa3ec1504c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800c4f0 tx=0x7fa3e800c800 comp rx=0 tx=0).stop 2024-09-16T07:25:05.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f37fe700 1 -- 172.21.15.37:0/713333870 reap_dead start 2024-09-16T07:25:05.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 shutdown_connections 2024-09-16T07:25:05.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 >> 172.21.15.37:0/713333870 conn(0x7fa3ec006440 msgr2=0x7fa3ec023780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:05.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 shutdown_connections 2024-09-16T07:25:05.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.618+0000 7fa3f3fff700 1 -- 172.21.15.37:0/713333870 wait complete. 2024-09-16T07:25:05.633 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T07:25:05.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient: get_monmap_and_config 2024-09-16T07:25:05.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient: build_initial_monmap 2024-09-16T07:25:05.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient: monmap: 2024-09-16T07:25:05.757 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:05.757 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:05.756588+0000 2024-09-16T07:25:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:05.756588+0000 2024-09-16T07:25:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:05.758 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 1 Processor -- start 2024-09-16T07:25:05.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 1 -- start start 2024-09-16T07:25:05.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient: init 2024-09-16T07:25:05.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:05.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-16T07:25:05.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac540b63f0 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:05.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fac540234f0 0x7fac540be850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient(hunting): picked mon.noname-c con 0x7fac540234f0 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:05.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fac540bed90 0x7fac540c1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient(hunting): picked mon.noname-a con 0x7fac540bed90 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:05.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac540b6030 con 0x7fac540b63f0 2024-09-16T07:25:05.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac54020000 con 0x7fac540bed90 2024-09-16T07:25:05.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5401fce0 con 0x7fac540234f0 2024-09-16T07:25:05.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:05.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient(hunting): authenticate will time out at 1226.242470s 2024-09-16T07:25:05.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac60883700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fac540b63f0 0x7fac540b67d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54504/0 (socket says 172.21.15.37:54504) 2024-09-16T07:25:05.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fac540bed90 0x7fac540c1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac60883700 1 -- 172.21.15.37:0/168889398 learned_addr learned my addr 172.21.15.37:0/168889398 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:05.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 10 monclient(hunting): get_auth_request con 0x7fac540bed90 auth_method 0 2024-09-16T07:25:05.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5bfff700 1 --2- 172.21.15.37:0/168889398 >> v2:172.21.15.143:3300/0 conn(0x7fac540234f0 0x7fac540be850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5bfff700 10 monclient(hunting): get_auth_request con 0x7fac540234f0 auth_method 0 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 10 monclient(hunting): handle_auth_done global_id 54228 payload 274 2024-09-16T07:25:05.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 1 -- 172.21.15.37:0/168889398 >> v2:172.21.15.143:3300/0 conn(0x7fac540234f0 msgr2=0x7fac540be850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 1 --2- 172.21.15.37:0/168889398 >> v2:172.21.15.143:3300/0 conn(0x7fac540234f0 0x7fac540be850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 1 -- 172.21.15.37:0/168889398 >> v1:172.21.15.37:6789/0 conn(0x7fac540b63f0 legacy=0x7fac540b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 10 monclient: _finish_hunting 0 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 1 monclient: found mon.noname-a 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 1 -- 172.21.15.37:0/168889398 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac54023940 con 0x7fac540bed90 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 1 -- 172.21.15.37:0/168889398 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024157666 0 0) 0x7fac540b6030 con 0x7fac540b63f0 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61084700 1 --2- 172.21.15.37:0/168889398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac540bed90 0x7fac540c1180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fac5c068a10 tx=0x7fac5c066f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0806c98bcdcf5f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 1 -- 172.21.15.37:0/168889398 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c06a310 con 0x7fac540bed90 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 10 monclient: dump: 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 10 monclient: _finish_auth 0 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 1 -- 172.21.15.37:0/168889398 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c08e070 con 0x7fac540bed90 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 5 monclient: authenticate success, global_id 54228 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 1 -- 172.21.15.37:0/168889398 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c067280 con 0x7fac540bed90 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac61885700 4 set_mon_vals no callback set 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.754+0000 7fac5b7fe700 10 monclient: dump: 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.775 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient: shutdown 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 -- 172.21.15.37:0/168889398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac540bed90 msgr2=0x7fac540c1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 --2- 172.21.15.37:0/168889398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac540bed90 0x7fac540c1180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fac5c068a10 tx=0x7fac5c066f10 comp rx=0 tx=0).stop 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 -- 172.21.15.37:0/168889398 shutdown_connections 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 --2- 172.21.15.37:0/168889398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac540bed90 0x7fac540c1180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 --2- 172.21.15.37:0/168889398 >> v2:172.21.15.143:3300/0 conn(0x7fac540234f0 0x7fac540be850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 -- 172.21.15.37:0/168889398 >> 172.21.15.37:0/168889398 conn(0x7fac540064b0 msgr2=0x7fac540068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 -- 172.21.15.37:0/168889398 shutdown_connections 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 -- 172.21.15.37:0/168889398 wait complete. 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient: build_initial_monmap 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient: monmap: 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:05.777 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:05.778 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:05.778 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:05.778 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:05.778 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 Processor -- start 2024-09-16T07:25:05.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 -- start start 2024-09-16T07:25:05.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient: init 2024-09-16T07:25:05.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:05.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:25:05.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac540234f0 0x7fac540b5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient(hunting): picked mon.noname-a con 0x7fac540234f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac540b63f0 0x7fac540b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient(hunting): picked mon.noname-b con 0x7fac540b63f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac540bed90 0x7fac540bc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient(hunting): picked mon.noname-c con 0x7fac540bed90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac540a1840 con 0x7fac540234f0 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac540a0350 con 0x7fac540b63f0 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac54020870 con 0x7fac540bed90 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61885700 10 monclient(hunting): authenticate will time out at 1226.247466s 2024-09-16T07:25:05.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61084700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac540bed90 0x7fac540bc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac60883700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac540234f0 0x7fac540b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61084700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac540bed90 0x7fac540bc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35070/0 (socket says 172.21.15.37:35070) 2024-09-16T07:25:05.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61084700 1 -- 172.21.15.37:0/2589348177 learned_addr learned my addr 172.21.15.37:0/2589348177 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:05.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61084700 10 monclient(hunting): get_auth_request con 0x7fac540bed90 auth_method 0 2024-09-16T07:25:05.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61084700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61084700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61084700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac60883700 10 monclient(hunting): get_auth_request con 0x7fac540234f0 auth_method 0 2024-09-16T07:25:05.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac60883700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac60883700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac60883700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61084700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61084700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac61084700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac60883700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:05.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac60883700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:05.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac60883700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:05.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac5bfff700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac540b63f0 0x7fac540b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac5bfff700 10 monclient(hunting): get_auth_request con 0x7fac540b63f0 auth_method 0 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.758+0000 7fac5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 10 monclient(hunting): handle_auth_done global_id 44603 payload 824 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 1 -- 172.21.15.37:0/2589348177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac540b63f0 msgr2=0x7fac540b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac540b63f0 0x7fac540b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 1 -- 172.21.15.37:0/2589348177 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac540234f0 msgr2=0x7fac540b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac540234f0 0x7fac540b5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 10 monclient: _finish_hunting 0 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 1 monclient: found mon.noname-c 2024-09-16T07:25:05.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:05.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:25:05.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 1 -- 172.21.15.37:0/2589348177 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c04f5c0 con 0x7fac540bed90 2024-09-16T07:25:05.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61084700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac540bed90 0x7fac540bc2b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fac5c067a20 tx=0x7fac5c06a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 1 -- 172.21.15.37:0/2589348177 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c067280 con 0x7fac540bed90 2024-09-16T07:25:05.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:05.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 10 monclient: dump: 2024-09-16T07:25:05.784 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.784 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.784 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.784 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.784 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 10 monclient: _finish_auth 0 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 10 monclient: _check_auth_tickets 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 5 monclient: authenticate success, global_id 44603 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 1 -- 172.21.15.37:0/2589348177 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c06bbb0 con 0x7fac540bed90 2024-09-16T07:25:05.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 10 monclient: _renew_subs 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 1 -- 172.21.15.37:0/2589348177 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5414f150 con 0x7fac540bed90 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 1 -- 172.21.15.37:0/2589348177 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c073850 con 0x7fac540bed90 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 10 monclient: dump: 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.786 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:05.787 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:05.787 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:05.787 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:05.787 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:05.787 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:05.787 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:05.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 10 monclient: _renew_subs 2024-09-16T07:25:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac5affd700 4 set_mon_vals no callback set 2024-09-16T07:25:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 1 -- 172.21.15.37:0/2589348177 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac540bde60 con 0x7fac540bed90 2024-09-16T07:25:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 1 -- 172.21.15.37:0/2589348177 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fac5c0694a0 con 0x7fac540bed90 2024-09-16T07:25:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fac40035600 0x7fac40037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:05.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:05.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:05.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac597fa700 1 -- 172.21.15.37:0/2589348177 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fac5c0c2b30 con 0x7fac540bed90 2024-09-16T07:25:05.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac61885700 1 -- 172.21.15.37:0/2589348177 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5414f150 con 0x7fac540bed90 2024-09-16T07:25:05.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac60883700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fac40035600 0x7fac40037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:05.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac60883700 10 monclient: get_auth_request con 0x7fac40035600 auth_method 0 2024-09-16T07:25:05.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.762+0000 7fac60883700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fac40035600 0x7fac40037ac0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fac48000f60 tx=0x7fac48007e00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:05.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.766+0000 7fac597fa700 1 -- 172.21.15.37:0/2589348177 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fac5414f150 con 0x7fac540bed90 2024-09-16T07:25:05.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.766+0000 7fac597fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:05.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.766+0000 7fac597fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:05.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 -- 172.21.15.37:0/2589348177 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fac540b6ee0 con 0x7fac40035600 2024-09-16T07:25:05.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac597fa700 1 -- 172.21.15.37:0/2589348177 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fac48001370 con 0x7fac40035600 2024-09-16T07:25:05.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 -- 172.21.15.37:0/2589348177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fac40035600 msgr2=0x7fac40037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fac40035600 0x7fac40037ac0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fac48000f60 tx=0x7fac48007e00 comp rx=0 tx=0).stop 2024-09-16T07:25:05.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 10 monclient: shutdown 2024-09-16T07:25:05.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:05.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 -- 172.21.15.37:0/2589348177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac540bed90 msgr2=0x7fac540bc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:05.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac540bed90 0x7fac540bc2b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fac5c067a20 tx=0x7fac5c06a0b0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 -- 172.21.15.37:0/2589348177 shutdown_connections 2024-09-16T07:25:05.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fac40035600 0x7fac40037ac0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac540bed90 0x7fac540bc2b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac540b63f0 0x7fac540b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 --2- 172.21.15.37:0/2589348177 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac540234f0 0x7fac540b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:05.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 -- 172.21.15.37:0/2589348177 >> 172.21.15.37:0/2589348177 conn(0x7fac540064b0 msgr2=0x7fac54027d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:05.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 -- 172.21.15.37:0/2589348177 shutdown_connections 2024-09-16T07:25:05.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:05.942+0000 7fac61885700 1 -- 172.21.15.37:0/2589348177 wait complete. 2024-09-16T07:25:05.949 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T07:25:05.960 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T07:25:05.960 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T07:25:06.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient: get_monmap_and_config 2024-09-16T07:25:06.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient: build_initial_monmap 2024-09-16T07:25:06.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient: monmap: 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:06.076663+0000 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:06.076663+0000 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:06.078 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 1 Processor -- start 2024-09-16T07:25:06.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 1 -- start start 2024-09-16T07:25:06.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient: init 2024-09-16T07:25:06.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:06.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T07:25:06.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45b009f3c0 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:06.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f45b009f7a0 0x7f45b009fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient(hunting): picked mon.noname-a con 0x7f45b009f7a0 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:06.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45b01373c0 addr v1:172.21.15.143:6789/0 2024-09-16T07:25:06.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b009d7b0 con 0x7f45b009f3c0 2024-09-16T07:25:06.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b009ad10 con 0x7f45b01373c0 2024-09-16T07:25:06.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b009df50 con 0x7f45b009f7a0 2024-09-16T07:25:06.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:06.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient(hunting): authenticate will time out at 1226.562524s 2024-09-16T07:25:06.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f45b009f7a0 0x7f45b009fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f45b009f7a0 0x7f45b009fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39514/0 (socket says 172.21.15.37:39514) 2024-09-16T07:25:06.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 1 -- 172.21.15.37:0/783644628 learned_addr learned my addr 172.21.15.37:0/783644628 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:06.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 10 monclient(hunting): get_auth_request con 0x7f45b009f7a0 auth_method 0 2024-09-16T07:25:06.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45af7fe700 1 --1- 172.21.15.37:0/783644628 >> v1:172.21.15.37:6789/0 conn(0x7f45b009f3c0 0x7f45b009d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54514/0 (socket says 172.21.15.37:54514) 2024-09-16T07:25:06.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 10 monclient(hunting): handle_auth_done global_id 54237 payload 274 2024-09-16T07:25:06.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 1 -- 172.21.15.37:0/783644628 >> v1:172.21.15.143:6789/0 conn(0x7f45b01373c0 legacy=0x7f45b01397b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:06.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 1 -- 172.21.15.37:0/783644628 >> v1:172.21.15.37:6789/0 conn(0x7f45b009f3c0 legacy=0x7f45b009d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 10 monclient: _finish_hunting 0 2024-09-16T07:25:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 1 monclient: found mon.noname-a 2024-09-16T07:25:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 1 -- 172.21.15.37:0/783644628 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b009bc50 con 0x7f45b009f7a0 2024-09-16T07:25:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45aeffd700 1 --2- 172.21.15.37:0/783644628 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45b009f7a0 0x7f45b009fbb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f45a000bbb0 tx=0x7f45a000bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee56407987f0963b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:06.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 1 -- 172.21.15.37:0/783644628 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a000dd50 con 0x7f45b009f7a0 2024-09-16T07:25:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 10 monclient: dump: 2024-09-16T07:25:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.087 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.088 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 10 monclient: _finish_auth 0 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 10 monclient: _check_auth_tickets 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 1 -- 172.21.15.37:0/783644628 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a0004030 con 0x7f45b009f7a0 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 5 monclient: authenticate success, global_id 54237 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:06.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 1 -- 172.21.15.37:0/783644628 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a00043e0 con 0x7f45b009f7a0 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45b4c9c700 4 set_mon_vals no callback set 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.074+0000 7f45ae7fc700 10 monclient: dump: 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.090 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 10 monclient: shutdown 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 1 -- 172.21.15.37:0/783644628 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45b009f7a0 msgr2=0x7f45b009fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 1 --2- 172.21.15.37:0/783644628 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45b009f7a0 0x7f45b009fbb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f45a000bbb0 tx=0x7f45a000bec0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 1 -- 172.21.15.37:0/783644628 shutdown_connections 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 1 --2- 172.21.15.37:0/783644628 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45b009f7a0 0x7f45b009fbb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 1 -- 172.21.15.37:0/783644628 >> 172.21.15.37:0/783644628 conn(0x7f45b00a04b0 msgr2=0x7f45b00a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 1 -- 172.21.15.37:0/783644628 shutdown_connections 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 1 -- 172.21.15.37:0/783644628 wait complete. 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 10 monclient: build_initial_monmap 2024-09-16T07:25:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 10 monclient: monmap: 2024-09-16T07:25:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:06.092 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:06.092 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 1 Processor -- start 2024-09-16T07:25:06.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 1 -- start start 2024-09-16T07:25:06.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 10 monclient: init 2024-09-16T07:25:06.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.078+0000 7f45b4c9c700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:06.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:25:06.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45b009f3c0 0x7f45b011d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient(hunting): picked mon.noname-a con 0x7f45b009f3c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:06.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45b009f7a0 0x7f45b011dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient(hunting): picked mon.noname-b con 0x7f45b009f7a0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:06.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45b01373c0 0x7f45b011e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient(hunting): picked mon.noname-c con 0x7f45b01373c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b00725f0 con 0x7f45b009f3c0 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b0121450 con 0x7f45b009f7a0 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b011f600 con 0x7f45b01373c0 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient(hunting): authenticate will time out at 1226.568213s 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45b009f3c0 0x7f45b011d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45b009f3c0 0x7f45b011d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39528/0 (socket says 172.21.15.37:39528) 2024-09-16T07:25:06.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45b01373c0 0x7f45b011e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 1 -- 172.21.15.37:0/3921088373 learned_addr learned my addr 172.21.15.37:0/3921088373 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 10 monclient(hunting): get_auth_request con 0x7f45b009f3c0 auth_method 0 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 10 monclient(hunting): get_auth_request con 0x7f45b01373c0 auth_method 0 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45aeffd700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45b009f7a0 0x7f45b011dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45aeffd700 10 monclient(hunting): get_auth_request con 0x7f45b009f7a0 auth_method 0 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45aeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45aeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45aeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45aeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 10 monclient(hunting): handle_auth_done global_id 44606 payload 824 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 1 -- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45b009f7a0 msgr2=0x7f45b011dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45b009f7a0 0x7f45b011dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 1 -- 172.21.15.37:0/3921088373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45b009f3c0 msgr2=0x7f45b011d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45b009f3c0 0x7f45b011d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 10 monclient: _finish_hunting 0 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 1 monclient: found mon.noname-c 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 1 -- 172.21.15.37:0/3921088373 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a000b810 con 0x7f45b01373c0 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45affff700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45b01373c0 0x7f45b011e140 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f45a800cb40 tx=0x7f45a800ce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 1 -- 172.21.15.37:0/3921088373 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a800a9a0 con 0x7f45b01373c0 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 10 monclient: dump: 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.108 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 10 monclient: _finish_auth 0 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 10 monclient: _check_auth_tickets 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 1 -- 172.21.15.37:0/3921088373 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a800ab00 con 0x7f45b01373c0 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 5 monclient: authenticate success, global_id 44606 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient: _renew_subs 2024-09-16T07:25:06.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b0136560 con 0x7f45b01373c0 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 1 -- 172.21.15.37:0/3921088373 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8012420 con 0x7f45b01373c0 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 10 monclient: dump: 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.110 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45adffb700 4 set_mon_vals no callback set 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient: _renew_subs 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b0121170 con 0x7f45b01373c0 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 1 -- 172.21.15.37:0/3921088373 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f45a8012580 con 0x7f45b01373c0 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f45b00a00f0 con 0x7f45b01373c0 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f459c035600 0x7f459c037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 1 -- 172.21.15.37:0/3921088373 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f45a8048110 con 0x7f45b01373c0 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] conn(0x7f459c03b410 0x7f459c03d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 1 -- 172.21.15.37:0/3921088373 --> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f459c03dee0 con 0x7f459c03b410 2024-09-16T07:25:06.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 1 -- 172.21.15.37:0/3921088373 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45a80228f0 con 0x7f45b01373c0 2024-09-16T07:25:06.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f4597fff700 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-16T07:25:06.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f459c035600 0x7f459c037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.082+0000 7f45af7fe700 10 monclient: get_auth_request con 0x7f459c035600 auth_method 0 2024-09-16T07:25:06.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.086+0000 7f45aeffd700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] conn(0x7f459c03b410 0x7f459c03d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.086+0000 7f45aeffd700 10 monclient: get_auth_request con 0x7f459c03b410 auth_method 0 2024-09-16T07:25:06.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.086+0000 7f45af7fe700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f459c035600 0x7f459c037ac0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f45a4004e90 tx=0x7f45a4001120 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:06.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.086+0000 7f45aeffd700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] conn(0x7f459c03b410 0x7f459c03d830 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:06.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.086+0000 7f4597fff700 1 -- 172.21.15.37:0/3921088373 <== osd.5 v2:172.21.15.143:6800/3921632375 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f459c03dee0 con 0x7f459c03b410 2024-09-16T07:25:06.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 --> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f45b0120d40 con 0x7f459c03b410 2024-09-16T07:25:06.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f4597fff700 1 -- 172.21.15.37:0/3921088373 <== osd.5 v2:172.21.15.143:6800/3921632375 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f45b0120d40 con 0x7f459c03b410 2024-09-16T07:25:06.113 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-16T07:25:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T07:25:06.113 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-16T07:25:06.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] conn(0x7f459c03b410 msgr2=0x7f459c03d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:6800/3921632375,v1:172.21.15.143:6801/3921632375] conn(0x7f459c03b410 0x7f459c03d830 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f459c035600 msgr2=0x7f459c037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f459c035600 0x7f459c037ac0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f45a4004e90 tx=0x7f45a4001120 comp rx=0 tx=0).stop 2024-09-16T07:25:06.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 10 monclient: shutdown 2024-09-16T07:25:06.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:06.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45b01373c0 msgr2=0x7f45b011e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 --2- 172.21.15.37:0/3921088373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45b01373c0 0x7f45b011e140 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f45a800cb40 tx=0x7f45a800ce50 comp rx=0 tx=0).stop 2024-09-16T07:25:06.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45affff700 1 -- 172.21.15.37:0/3921088373 reap_dead start 2024-09-16T07:25:06.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 shutdown_connections 2024-09-16T07:25:06.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 >> 172.21.15.37:0/3921088373 conn(0x7f45b00a04b0 msgr2=0x7f45b0070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:06.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 shutdown_connections 2024-09-16T07:25:06.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.106+0000 7f45b4c9c700 1 -- 172.21.15.37:0/3921088373 wait complete. 2024-09-16T07:25:06.124 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T07:25:06.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.234+0000 7fa0ca1de700 10 monclient: get_monmap_and_config 2024-09-16T07:25:06.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.234+0000 7fa0ca1de700 10 monclient: build_initial_monmap 2024-09-16T07:25:06.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.234+0000 7fa0ca1de700 10 monclient: monmap: 2024-09-16T07:25:06.239 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:06.238637+0000 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:06.238637+0000 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:06.240 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 Processor -- start 2024-09-16T07:25:06.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 -- start start 2024-09-16T07:25:06.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient: init 2024-09-16T07:25:06.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:06.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-16T07:25:06.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa0c40a0e90 0x7fa0c40a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient(hunting): picked mon.noname-a con 0x7fa0c40a0e90 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:06.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a1270 0x7fa0c40a2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient(hunting): picked mon.noname-b con 0x7fa0c40a1270 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:06.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0c40a2b80 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:06.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c40a36e0 con 0x7fa0c40a2b80 2024-09-16T07:25:06.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c406cf50 con 0x7fa0c40a0e90 2024-09-16T07:25:06.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c409f810 con 0x7fa0c40a1270 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient(hunting): authenticate will time out at 1226.724746s 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a1270 0x7fa0c40a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a1270 0x7fa0c40a2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35086/0 (socket says 172.21.15.37:35086) 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c91dc700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa0c40a0e90 0x7fa0c40a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 1 -- 172.21.15.37:0/2883025091 learned_addr learned my addr 172.21.15.37:0/2883025091 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 10 monclient(hunting): get_auth_request con 0x7fa0c40a1270 auth_method 0 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c91dc700 10 monclient(hunting): get_auth_request con 0x7fa0c40a0e90 auth_method 0 2024-09-16T07:25:06.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c91dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c91dc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c91dc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c91dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c91dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c91dc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 10 monclient(hunting): handle_auth_done global_id 44609 payload 274 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 1 -- 172.21.15.37:0/2883025091 >> v2:172.21.15.37:3300/0 conn(0x7fa0c40a0e90 msgr2=0x7fa0c40a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 1 --2- 172.21.15.37:0/2883025091 >> v2:172.21.15.37:3300/0 conn(0x7fa0c40a0e90 0x7fa0c40a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 1 -- 172.21.15.37:0/2883025091 >> v1:172.21.15.37:6789/0 conn(0x7fa0c40a2b80 legacy=0x7fa0c40a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 10 monclient: _finish_hunting 0 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 1 monclient: found mon.noname-b 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 1 -- 172.21.15.37:0/2883025091 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c409f250 con 0x7fa0c40a1270 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0c89db700 1 --2- 172.21.15.37:0/2883025091 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a1270 0x7fa0c40a2640 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4009910 tx=0x7fa0b4009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8041977da37d51e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 1 -- 172.21.15.37:0/2883025091 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4011830 con 0x7fa0c40a1270 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 10 monclient: dump: 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.248 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 10 monclient: _finish_auth 0 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 10 monclient: _check_auth_tickets 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 5 monclient: authenticate success, global_id 44609 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:06.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 1 -- 172.21.15.37:0/2883025091 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b4011990 con 0x7fa0c40a1270 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 4 set_mon_vals no callback set 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 1 -- 172.21.15.37:0/2883025091 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4011d40 con 0x7fa0c40a1270 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0bbfff700 10 monclient: dump: 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.250 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 10 monclient: shutdown 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 -- 172.21.15.37:0/2883025091 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a1270 msgr2=0x7fa0c40a2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 --2- 172.21.15.37:0/2883025091 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a1270 0x7fa0c40a2640 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4009910 tx=0x7fa0b4009c20 comp rx=0 tx=0).stop 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 -- 172.21.15.37:0/2883025091 shutdown_connections 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 --2- 172.21.15.37:0/2883025091 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a1270 0x7fa0c40a2640 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 --2- 172.21.15.37:0/2883025091 >> v2:172.21.15.37:3300/0 conn(0x7fa0c40a0e90 0x7fa0c40a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 -- 172.21.15.37:0/2883025091 >> 172.21.15.37:0/2883025091 conn(0x7fa0c40a4740 msgr2=0x7fa0c40a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 -- 172.21.15.37:0/2883025091 shutdown_connections 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.238+0000 7fa0ca1de700 1 -- 172.21.15.37:0/2883025091 wait complete. 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient: build_initial_monmap 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient: monmap: 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:06.252 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:06.253 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:06.253 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 1 Processor -- start 2024-09-16T07:25:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 1 -- start start 2024-09-16T07:25:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient: init 2024-09-16T07:25:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:06.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0c40a0e90 0x7fa0c413bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient(hunting): picked mon.noname-a con 0x7fa0c40a0e90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c40a1270 0x7fa0c413c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient(hunting): picked mon.noname-b con 0x7fa0c40a1270 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a2b80 0x7fa0c4122c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient(hunting): picked mon.noname-c con 0x7fa0c40a2b80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c413c980 con 0x7fa0c40a0e90 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c41268c0 con 0x7fa0c40a1270 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c4125410 con 0x7fa0c40a2b80 2024-09-16T07:25:06.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient(hunting): authenticate will time out at 1226.729924s 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c91dc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0c40a0e90 0x7fa0c413bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c91dc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0c40a0e90 0x7fa0c413bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39550/0 (socket says 172.21.15.37:39550) 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a2b80 0x7fa0c4122c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c91dc700 1 -- 172.21.15.37:0/3204438031 learned_addr learned my addr 172.21.15.37:0/3204438031 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c91dc700 10 monclient(hunting): get_auth_request con 0x7fa0c40a0e90 auth_method 0 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c91dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c91dc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c91dc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 10 monclient(hunting): get_auth_request con 0x7fa0c40a2b80 auth_method 0 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c91dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c91dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c91dc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c89db700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c40a1270 0x7fa0c413c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c89db700 10 monclient(hunting): get_auth_request con 0x7fa0c40a1270 auth_method 0 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c89db700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c89db700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c89db700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c89db700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c89db700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c89db700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 10 monclient(hunting): handle_auth_done global_id 44612 payload 824 2024-09-16T07:25:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 1 -- 172.21.15.37:0/3204438031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c40a1270 msgr2=0x7fa0c413c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c40a1270 0x7fa0c413c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 1 -- 172.21.15.37:0/3204438031 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0c40a0e90 msgr2=0x7fa0c413bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0c40a0e90 0x7fa0c413bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 10 monclient: _finish_hunting 0 2024-09-16T07:25:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 1 monclient: found mon.noname-c 2024-09-16T07:25:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:06.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 1 -- 172.21.15.37:0/3204438031 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b4009570 con 0x7fa0c40a2b80 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0c99dd700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a2b80 0x7fa0c4122c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00c400 tx=0x7fa0bc00c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 1 -- 172.21.15.37:0/3204438031 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0052d0 con 0x7fa0c40a2b80 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 10 monclient: dump: 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.261 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 10 monclient: _finish_auth 0 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 10 monclient: _check_auth_tickets 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 5 monclient: authenticate success, global_id 44612 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 1 -- 172.21.15.37:0/3204438031 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc005430 con 0x7fa0c40a2b80 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 1 -- 172.21.15.37:0/3204438031 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0055a0 con 0x7fa0c40a2b80 2024-09-16T07:25:06.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient: _renew_subs 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0bb7fe700 4 set_mon_vals no callback set 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 1 -- 172.21.15.37:0/3204438031 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c409dfc0 con 0x7fa0c40a2b80 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0b9ffb700 10 monclient: dump: 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.263 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient: _renew_subs 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.242+0000 7fa0ca1de700 1 -- 172.21.15.37:0/3204438031 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c41265e0 con 0x7fa0c40a2b80 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.246+0000 7fa0b9ffb700 1 -- 172.21.15.37:0/3204438031 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fa0bc003900 con 0x7fa0c40a2b80 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.246+0000 7fa0b9ffb700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa0a8035600 0x7fa0a8037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.246+0000 7fa0ca1de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.246+0000 7fa0b9ffb700 1 -- 172.21.15.37:0/3204438031 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fa0bc046d00 con 0x7fa0c40a2b80 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.246+0000 7fa0ca1de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:06.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.246+0000 7fa0ca1de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:06.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.246+0000 7fa0ca1de700 1 -- 172.21.15.37:0/3204438031 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c40a2110 con 0x7fa0c40a2b80 2024-09-16T07:25:06.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.246+0000 7fa0c91dc700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa0a8035600 0x7fa0a8037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.246+0000 7fa0c91dc700 10 monclient: get_auth_request con 0x7fa0a8035600 auth_method 0 2024-09-16T07:25:06.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.246+0000 7fa0c91dc700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa0a8035600 0x7fa0a8037ac0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009a40 tx=0x7fa0c000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:06.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.254+0000 7fa0b9ffb700 1 -- 172.21.15.37:0/3204438031 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa0bc017020 con 0x7fa0c40a2b80 2024-09-16T07:25:06.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.254+0000 7fa0b9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:06.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.254+0000 7fa0b9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:06.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 -- 172.21.15.37:0/3204438031 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0c40a16a0 con 0x7fa0a8035600 2024-09-16T07:25:06.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0b9ffb700 1 -- 172.21.15.37:0/3204438031 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa0c000f040 con 0x7fa0a8035600 2024-09-16T07:25:06.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 -- 172.21.15.37:0/3204438031 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa0a8035600 msgr2=0x7fa0a8037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa0a8035600 0x7fa0a8037ac0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009a40 tx=0x7fa0c000b040 comp rx=0 tx=0).stop 2024-09-16T07:25:06.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 10 monclient: shutdown 2024-09-16T07:25:06.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:06.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 -- 172.21.15.37:0/3204438031 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a2b80 msgr2=0x7fa0c4122c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a2b80 0x7fa0c4122c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00c400 tx=0x7fa0bc00c710 comp rx=0 tx=0).stop 2024-09-16T07:25:06.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 -- 172.21.15.37:0/3204438031 shutdown_connections 2024-09-16T07:25:06.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fa0a8035600 0x7fa0a8037ac0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0c40a2b80 0x7fa0c4122c40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0c40a1270 0x7fa0c413c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 --2- 172.21.15.37:0/3204438031 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0c40a0e90 0x7fa0c413bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 -- 172.21.15.37:0/3204438031 >> 172.21.15.37:0/3204438031 conn(0x7fa0c40a4740 msgr2=0x7fa0c409e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:06.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.422+0000 7fa0ca1de700 1 -- 172.21.15.37:0/3204438031 shutdown_connections 2024-09-16T07:25:06.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.426+0000 7fa0ca1de700 1 -- 172.21.15.37:0/3204438031 wait complete. 2024-09-16T07:25:06.429 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T07:25:06.440 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient: get_monmap_and_config 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient: build_initial_monmap 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient: monmap: 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:06.550093+0000 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:06.550093+0000 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:06.552 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 1 Processor -- start 2024-09-16T07:25:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 1 -- start start 2024-09-16T07:25:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient: init 2024-09-16T07:25:06.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:06.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-16T07:25:06.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86180a2750 addr v1:172.21.15.37:6789/0 2024-09-16T07:25:06.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86180a3240 addr v1:172.21.15.143:6789/0 2024-09-16T07:25:06.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 0x7f861813ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient(hunting): picked mon.noname-b con 0x7f861809d580 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:06.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861809c6c0 con 0x7f86180a2750 2024-09-16T07:25:06.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861809c9a0 con 0x7f86180a3240 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861809b130 con 0x7f861809d580 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.546+0000 7f8617fff700 10 monclient(hunting): authenticate will time out at 1227.035910s 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 0x7f861813ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8616ffd700 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f86180a2750 0x7f86180a2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54546/0 (socket says 172.21.15.37:54546) 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8616ffd700 1 -- 172.21.15.37:0/1839753395 learned_addr learned my addr 172.21.15.37:0/1839753395 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 10 monclient(hunting): get_auth_request con 0x7f861809d580 auth_method 0 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 10 monclient(hunting): handle_auth_done global_id 44615 payload 274 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 1 -- 172.21.15.37:0/1839753395 >> v1:172.21.15.143:6789/0 conn(0x7f86180a3240 legacy=0x7f8618137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 1 -- 172.21.15.37:0/1839753395 >> v1:172.21.15.37:6789/0 conn(0x7f86180a2750 legacy=0x7f86180a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 10 monclient: _finish_hunting 0 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 1 monclient: found mon.noname-b 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 1 -- 172.21.15.37:0/1839753395 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861809c3e0 con 0x7f861809d580 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86177fe700 1 --2- 172.21.15.37:0/1839753395 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 0x7f861813ab90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8610009df0 tx=0x7f861000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a709101eebe13e7d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 1 -- 172.21.15.37:0/1839753395 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861000e070 con 0x7f861809d580 2024-09-16T07:25:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T07:25:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 10 monclient: dump: 2024-09-16T07:25:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 10 monclient: _finish_auth 0 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 10 monclient: _check_auth_tickets 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 1 -- 172.21.15.37:0/1839753395 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861000bb90 con 0x7f861809d580 2024-09-16T07:25:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 5 monclient: authenticate success, global_id 44615 2024-09-16T07:25:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 1 -- 172.21.15.37:0/1839753395 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8610012430 con 0x7f861809d580 2024-09-16T07:25:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 4 set_mon_vals no callback set 2024-09-16T07:25:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f86167fc700 10 monclient: dump: 2024-09-16T07:25:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.563 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 10 monclient: shutdown 2024-09-16T07:25:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 1 -- 172.21.15.37:0/1839753395 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 msgr2=0x7f861813ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 1 --2- 172.21.15.37:0/1839753395 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 0x7f861813ab90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8610009df0 tx=0x7f861000b330 comp rx=0 tx=0).stop 2024-09-16T07:25:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 1 -- 172.21.15.37:0/1839753395 shutdown_connections 2024-09-16T07:25:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 1 --2- 172.21.15.37:0/1839753395 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 0x7f861813ab90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 1 -- 172.21.15.37:0/1839753395 >> 172.21.15.37:0/1839753395 conn(0x7f86180a3ea0 msgr2=0x7f86180a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 1 -- 172.21.15.37:0/1839753395 shutdown_connections 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 1 -- 172.21.15.37:0/1839753395 wait complete. 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.550+0000 7f8617fff700 10 monclient: build_initial_monmap 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient: monmap: 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:06.565 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 1 Processor -- start 2024-09-16T07:25:06.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 1 -- start start 2024-09-16T07:25:06.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient: init 2024-09-16T07:25:06.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:06.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:25:06.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 0x7f86181c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient(hunting): picked mon.noname-c con 0x7f861809d580 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86180a2750 0x7f86181c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient(hunting): picked mon.noname-b con 0x7f86180a2750 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86180a3240 0x7f86181219a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient(hunting): picked mon.noname-a con 0x7f86180a3240 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86181245f0 con 0x7f86180a3240 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8618124840 con 0x7f86180a2750 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86181c9090 con 0x7f861809d580 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient(hunting): authenticate will time out at 1227.041726s 2024-09-16T07:25:06.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86180a3240 0x7f86181219a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8616ffd700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 0x7f86181c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86180a3240 0x7f86181219a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:39554/0 (socket says 172.21.15.37:39554) 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 1 -- 172.21.15.37:0/3668492337 learned_addr learned my addr 172.21.15.37:0/3668492337 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 10 monclient(hunting): get_auth_request con 0x7f86180a3240 auth_method 0 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8616ffd700 10 monclient(hunting): get_auth_request con 0x7f861809d580 auth_method 0 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8616ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8616ffd700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8616ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f860e7fc700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86180a2750 0x7f86181c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8616ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:06.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8616ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:06.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8616ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:06.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f860e7fc700 10 monclient(hunting): get_auth_request con 0x7f86180a2750 auth_method 0 2024-09-16T07:25:06.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f860e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:06.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f860e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:06.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f860e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:06.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 10 monclient(hunting): handle_auth_done global_id 54258 payload 824 2024-09-16T07:25:06.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 1 -- 172.21.15.37:0/3668492337 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 msgr2=0x7f86181c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 0x7f86181c8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 1 -- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86180a2750 msgr2=0x7f86181c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86180a2750 0x7f86181c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 10 monclient: _finish_hunting 0 2024-09-16T07:25:06.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 1 monclient: found mon.noname-a 2024-09-16T07:25:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 1 -- 172.21.15.37:0/3668492337 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8610009a70 con 0x7f86180a3240 2024-09-16T07:25:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f860e7fc700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86180a2750 0x7f86181c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:25:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f86177fe700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86180a3240 0x7f86181219a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f861000b370 tx=0x7f8610004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 1 -- 172.21.15.37:0/3668492337 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8610012ce0 con 0x7f86180a3240 2024-09-16T07:25:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:06.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 10 monclient: dump: 2024-09-16T07:25:06.575 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.575 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.575 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.575 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.575 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 10 monclient: _finish_auth 0 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 10 monclient: _check_auth_tickets 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 5 monclient: authenticate success, global_id 54258 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 1 -- 172.21.15.37:0/3668492337 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8610020070 con 0x7f86180a3240 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient: _renew_subs 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 1 -- 172.21.15.37:0/3668492337 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86180724c0 con 0x7f86180a3240 2024-09-16T07:25:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 1 -- 172.21.15.37:0/3668492337 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8610004280 con 0x7f86180a3240 2024-09-16T07:25:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8615ffb700 4 set_mon_vals no callback set 2024-09-16T07:25:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8614ff9700 10 monclient: dump: 2024-09-16T07:25:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient: _renew_subs 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.554+0000 7f8617fff700 1 -- 172.21.15.37:0/3668492337 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861806cb60 con 0x7f86180a3240 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.558+0000 7f8617fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.558+0000 7f8617fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.558+0000 7f8617fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:06.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.558+0000 7f8617fff700 1 -- 172.21.15.37:0/3668492337 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86180716c0 con 0x7f86180a3240 2024-09-16T07:25:06.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.558+0000 7f8614ff9700 1 -- 172.21.15.37:0/3668492337 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f86100043e0 con 0x7f86180a3240 2024-09-16T07:25:06.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.558+0000 7f8614ff9700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f85f8035600 0x7f85f8037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:06.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.558+0000 7f8614ff9700 1 -- 172.21.15.37:0/3668492337 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f8610022070 con 0x7f86180a3240 2024-09-16T07:25:06.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.558+0000 7f8616ffd700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f85f8035600 0x7f85f8037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:06.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.558+0000 7f8616ffd700 10 monclient: get_auth_request con 0x7f85f8035600 auth_method 0 2024-09-16T07:25:06.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.558+0000 7f8616ffd700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f85f8035600 0x7f85f8037ac0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8608005930 tx=0x7f8608009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:06.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.566+0000 7f8614ff9700 1 -- 172.21.15.37:0/3668492337 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8610012e60 con 0x7f86180a3240 2024-09-16T07:25:06.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.566+0000 7f8614ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:06.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.566+0000 7f8614ff9700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:06.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.738+0000 7f8617fff700 1 -- 172.21.15.37:0/3668492337 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f86181249b0 con 0x7f85f8035600 2024-09-16T07:25:06.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.742+0000 7f8614ff9700 1 -- 172.21.15.37:0/3668492337 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53161 (secure 0 0 0) 0x7f86181249b0 con 0x7f85f8035600 2024-09-16T07:25:06.746 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:25:06.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.742+0000 7f8617fff700 1 -- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f85f8035600 msgr2=0x7f85f8037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.742+0000 7f8617fff700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f85f8035600 0x7f85f8037ac0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8608005930 tx=0x7f8608009040 comp rx=0 tx=0).stop 2024-09-16T07:25:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.742+0000 7f8617fff700 10 monclient: shutdown 2024-09-16T07:25:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.742+0000 7f8617fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.742+0000 7f8617fff700 1 -- 172.21.15.37:0/3668492337 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86180a3240 msgr2=0x7f86181219a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.742+0000 7f8617fff700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86180a3240 0x7f86181219a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f861000b370 tx=0x7f8610004af0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.742+0000 7f8617fff700 1 -- 172.21.15.37:0/3668492337 shutdown_connections 2024-09-16T07:25:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.746+0000 7f8617fff700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86180a3240 0x7f86181219a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.746+0000 7f8617fff700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86180a2750 0x7f86181c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.746+0000 7f8617fff700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f85f8035600 0x7f85f8037ac0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.746+0000 7f8617fff700 1 --2- 172.21.15.37:0/3668492337 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f861809d580 0x7f86181c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.746+0000 7f8617fff700 1 -- 172.21.15.37:0/3668492337 >> 172.21.15.37:0/3668492337 conn(0x7f86180a3ea0 msgr2=0x7f861806cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.746+0000 7f8617fff700 1 -- 172.21.15.37:0/3668492337 shutdown_connections 2024-09-16T07:25:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:06.746+0000 7f8617fff700 1 -- 172.21.15.37:0/3668492337 wait complete. 2024-09-16T07:25:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T07:25:06.765 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-16T07:25:06.555834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":196056,"kb_used_data":25744,"kb_used_omap":9,"kb_used_meta":170230,"kb_avail":566034984,"statfs":{"total":579820584960,"available":579619823616,"internally_reserved":0,"allocated":26361856,"data_stored":10445776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":174316198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003202"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":283,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:40.676229+0000","last_change":"2024-09-16T07:18:57.472278+0000","last_active":"2024-09-16T07:24:40.676229+0000","last_peered":"2024-09-16T07:24:40.676229+0000","last_clean":"2024-09-16T07:24:40.676229+0000","last_became_active":"2024-09-16T07:18:55.463443+0000","last_became_peered":"2024-09-16T07:18:55.463443+0000","last_unstale":"2024-09-16T07:24:40.676229+0000","last_undegraded":"2024-09-16T07:24:40.676229+0000","last_fullsized":"2024-09-16T07:24:40.676229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059939100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":287,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:42.652909+0000","last_change":"2024-09-16T07:18:57.474462+0000","last_active":"2024-09-16T07:24:42.652909+0000","last_peered":"2024-09-16T07:24:42.652909+0000","last_clean":"2024-09-16T07:24:42.652909+0000","last_became_active":"2024-09-16T07:18:55.467864+0000","last_became_peered":"2024-09-16T07:18:55.467864+0000","last_unstale":"2024-09-16T07:24:42.652909+0000","last_undegraded":"2024-09-16T07:24:42.652909+0000","last_fullsized":"2024-09-16T07:24:42.652909+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066039799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":283,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:40.676160+0000","last_change":"2024-09-16T07:18:57.472359+0000","last_active":"2024-09-16T07:24:40.676160+0000","last_peered":"2024-09-16T07:24:40.676160+0000","last_clean":"2024-09-16T07:24:40.676160+0000","last_became_active":"2024-09-16T07:18:55.471978+0000","last_became_peered":"2024-09-16T07:18:55.471978+0000","last_unstale":"2024-09-16T07:24:40.676160+0000","last_undegraded":"2024-09-16T07:24:40.676160+0000","last_fullsized":"2024-09-16T07:24:40.676160+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077051799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":283,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.763092+0000","last_change":"2024-09-16T07:18:57.464442+0000","last_active":"2024-09-16T07:24:39.763092+0000","last_peered":"2024-09-16T07:24:39.763092+0000","last_clean":"2024-09-16T07:24:39.763092+0000","last_became_active":"2024-09-16T07:18:55.468584+0000","last_became_peered":"2024-09-16T07:18:55.468584+0000","last_unstale":"2024-09-16T07:24:39.763092+0000","last_undegraded":"2024-09-16T07:24:39.763092+0000","last_fullsized":"2024-09-16T07:24:39.763092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:20.380838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000689417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":291,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.768039+0000","last_change":"2024-09-16T07:18:57.470967+0000","last_active":"2024-09-16T07:24:39.768039+0000","last_peered":"2024-09-16T07:24:39.768039+0000","last_clean":"2024-09-16T07:24:39.768039+0000","last_became_active":"2024-09-16T07:18:55.472671+0000","last_became_peered":"2024-09-16T07:18:55.472671+0000","last_unstale":"2024-09-16T07:24:39.768039+0000","last_undegraded":"2024-09-16T07:24:39.768039+0000","last_fullsized":"2024-09-16T07:24:39.768039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:13.933764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000840762,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":298,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-16T07:24:39.767524+0000","last_change":"2024-09-16T07:18:57.740277+0000","last_active":"2024-09-16T07:24:39.767524+0000","last_peered":"2024-09-16T07:24:39.767524+0000","last_clean":"2024-09-16T07:24:39.767524+0000","last_became_active":"2024-09-16T07:18:55.469151+0000","last_became_peered":"2024-09-16T07:18:55.469151+0000","last_unstale":"2024-09-16T07:24:39.767524+0000","last_undegraded":"2024-09-16T07:24:39.767524+0000","last_fullsized":"2024-09-16T07:24:39.767524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:34.482147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064042899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":302,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:42.293038+0000","last_change":"2024-09-16T07:18:57.743569+0000","last_active":"2024-09-16T07:24:42.293038+0000","last_peered":"2024-09-16T07:24:42.293038+0000","last_clean":"2024-09-16T07:24:42.293038+0000","last_became_active":"2024-09-16T07:18:55.469565+0000","last_became_peered":"2024-09-16T07:18:55.469565+0000","last_unstale":"2024-09-16T07:24:42.293038+0000","last_undegraded":"2024-09-16T07:24:42.293038+0000","last_fullsized":"2024-09-16T07:24:42.293038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:46.360051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077454399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":284,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:40.676665+0000","last_change":"2024-09-16T07:18:57.472930+0000","last_active":"2024-09-16T07:24:40.676665+0000","last_peered":"2024-09-16T07:24:40.676665+0000","last_clean":"2024-09-16T07:24:40.676665+0000","last_became_active":"2024-09-16T07:18:55.464362+0000","last_became_peered":"2024-09-16T07:18:55.464362+0000","last_unstale":"2024-09-16T07:24:40.676665+0000","last_undegraded":"2024-09-16T07:24:40.676665+0000","last_fullsized":"2024-09-16T07:24:40.676665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:54.434573+0000","last_clean_scrub_stamp":"2024-09-16T07:18:54.434573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212957,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":432,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:24:51.428209+0000","last_change":"2024-09-16T07:18:49.375792+0000","last_active":"2024-09-16T07:24:51.428209+0000","last_peered":"2024-09-16T07:24:51.428209+0000","last_clean":"2024-09-16T07:24:51.428209+0000","last_became_active":"2024-09-16T07:18:49.373551+0000","last_became_peered":"2024-09-16T07:18:49.373551+0000","last_unstale":"2024-09-16T07:24:51.428209+0000","last_undegraded":"2024-09-16T07:24:51.428209+0000","last_fullsized":"2024-09-16T07:24:51.428209+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:18:47.386541+0000","last_clean_scrub_stamp":"2024-09-16T07:18:47.386541+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574927,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35232,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94336608,"statfs":{"total":96636764160,"available":96600686592,"internally_reserved":0,"allocated":4214784,"data_stored":1564647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.78800000000000003,"15min":0.83699999999999997},"min":{"1min":0.496,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1040000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.504},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.78800000000000003,"15min":0.80800000000000005},"min":{"1min":0.54400000000000004,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98799999999999999,"5min":1.329,"15min":1.329},"last":0.64100000000000001}]},{"osd":1,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.1779999999999999,"5min":2.177,"15min":2.177},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82599999999999996,"15min":0.872},"min":{"1min":0.58899999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.079,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.76400000000000001}]},{"osd":2,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.82699999999999996,"15min":0.82499999999999996},"min":{"1min":0.58899999999999997,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.091,"5min":1.23,"15min":1.23},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.92600000000000005,"15min":0.89300000000000002},"min":{"1min":0.60399999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.028,"5min":3.8610000000000002,"15min":3.8610000000000002},"last":0.60399999999999998}]},{"osd":4,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.73699999999999999,"15min":0.72799999999999998},"min":{"1min":0.47299999999999998,"5min":0.377,"15min":0.377},"max":{"1min":3.879,"5min":3.879,"15min":3.879},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.76900000000000002,"15min":0.78900000000000003},"min":{"1min":0.54900000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.7999999999999998,"5min":2.7999999999999998,"15min":2.7999999999999998},"last":0.80700000000000005}]},{"osd":5,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.74299999999999999,"15min":0.751},"min":{"1min":0.57499999999999996,"5min":0.432,"15min":0.432},"max":{"1min":1.05,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.91100000000000003,"15min":0.86699999999999999},"min":{"1min":0.69099999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.171,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.82899999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574927,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31136,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340704,"statfs":{"total":96636764160,"available":96604880896,"internally_reserved":0,"allocated":4214784,"data_stored":1564647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58099999999999996,"15min":0.57599999999999996},"min":{"1min":0.44700000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.145,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.65000000000000002,"15min":0.629},"min":{"1min":0.505,"5min":0.316,"15min":0.316},"max":{"1min":0.79800000000000004,"5min":1.135,"15min":1.135},"last":0.63600000000000001}]},{"osd":1,"last update":"Mon Sep 16 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.628,"15min":0.64800000000000002},"min":{"1min":0.52100000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.79800000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.747,"15min":0.69699999999999995},"min":{"1min":0.49099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.68600000000000005}]},{"osd":3,"last update":"Mon Sep 16 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79300000000000004,"15min":0.77400000000000002},"min":{"1min":0.57899999999999996,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.111,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.89800000000000002,"15min":0.86299999999999999},"min":{"1min":0.72599999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.242,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.76100000000000001}]},{"osd":4,"last update":"Mon Sep 16 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.92000000000000004,"15min":0.91300000000000003},"min":{"1min":0.67300000000000004,"5min":0.496,"15min":0.496},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84599999999999997,"15min":0.89400000000000002},"min":{"1min":0.63800000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.95999999999999996,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.90500000000000003}]},{"osd":5,"last update":"Mon Sep 16 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90400000000000003,"15min":0.91600000000000004},"min":{"1min":0.72499999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.177,"5min":1.419,"15min":1.419},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91100000000000003,"15min":0.93700000000000006},"min":{"1min":0.70199999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.284,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.78400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574927,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35236,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94336604,"statfs":{"total":96636764160,"available":96600682496,"internally_reserved":0,"allocated":4218880,"data_stored":1564666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 07:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65200000000000002,"15min":0.65600000000000003},"min":{"1min":0.48999999999999999,"5min":0.42099999999999999,"15min":0.41899999999999998},"max":{"1min":0.90900000000000003,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.60099999999999998}]},{"osd":2,"last update":"Mon Sep 16 07:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58399999999999996,"15min":0.626},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81799999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.374},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.70299999999999996,"15min":0.75},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.43099999999999999},"max":{"1min":0.86899999999999999,"5min":1.0720000000000001,"15min":1.2829999999999999},"last":0.55100000000000005}]},{"osd":3,"last update":"Mon Sep 16 07:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.85199999999999998,"15min":0.86199999999999999},"min":{"1min":0.60299999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.0669999999999999,"5min":1.181,"15min":1.611},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.84999999999999998,"15min":0.85399999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.95299999999999996,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.83399999999999996}]},{"osd":4,"last update":"Mon Sep 16 07:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.874,"15min":0.91100000000000003},"min":{"1min":0.76700000000000002,"5min":0.621,"15min":0.621},"max":{"1min":1.115,"5min":1.472,"15min":1.998},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.91300000000000003,"15min":0.91500000000000004},"min":{"1min":0.70999999999999996,"5min":0.63,"15min":0.63},"max":{"1min":1.2430000000000001,"5min":1.3080000000000001,"15min":1.4019999999999999},"last":0.86799999999999999}]},{"osd":5,"last update":"Mon Sep 16 07:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.89000000000000001,"15min":0.97799999999999998},"min":{"1min":0.66500000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2010000000000001,"5min":1.3420000000000001,"15min":1.673},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.97199999999999998,"15min":0.99199999999999999},"min":{"1min":0.68600000000000005,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.151,"5min":1.387,"15min":1.825},"last":1.012}]}]},{"osd":4,"up_from":12,"seq":51539607631,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31140,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340700,"statfs":{"total":96636764160,"available":96604876800,"internally_reserved":0,"allocated":4218880,"data_stored":1564666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.82699999999999996,"15min":0.84699999999999998},"min":{"1min":0.66900000000000004,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1060000000000001,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.84699999999999998,"15min":0.82599999999999996},"min":{"1min":0.70599999999999996,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2270000000000001,"5min":1.272,"15min":1.272},"last":0.79000000000000004}]},{"osd":1,"last update":"Mon Sep 16 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.80000000000000004,"15min":0.81799999999999995},"min":{"1min":0.60299999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.089,"5min":1.462,"15min":1.462},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.87,"5min":0.83399999999999996,"15min":0.86199999999999999},"min":{"1min":0.58599999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.268,"5min":1.391,"15min":1.548},"last":0.73699999999999999}]},{"osd":2,"last update":"Mon Sep 16 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.82199999999999995,"15min":0.78200000000000003},"min":{"1min":0.52200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.1819999999999999,"5min":2.246,"15min":2.246},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.86399999999999999,"15min":0.88800000000000001},"min":{"1min":0.66700000000000004,"5min":0.435,"15min":0.435},"max":{"1min":1.1759999999999999,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.86099999999999999}]},{"osd":3,"last update":"Mon Sep 16 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.72799999999999998,"15min":0.80800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85499999999999998,"5min":2.423,"15min":2.423},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.76500000000000001,"15min":0.78500000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95199999999999996,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.88500000000000001}]},{"osd":5,"last update":"Mon Sep 16 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80400000000000005,"15min":0.82599999999999996},"min":{"1min":0.47699999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.3740000000000001,"5min":1.415,"15min":1.8260000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.82999999999999996,"15min":0.86899999999999999},"min":{"1min":0.55400000000000005,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.1080000000000001,"5min":1.284,"15min":1.776},"last":0.84299999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607630,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":4747264,"data_stored":2093575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58199999999999996,"15min":0.57699999999999996},"min":{"1min":0.46500000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.97299999999999998,"5min":1.107,"15min":1.107},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63100000000000001,"15min":0.66300000000000003},"min":{"1min":0.51600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":1.23},"last":0.374}]},{"osd":2,"last update":"Mon Sep 16 07:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.64100000000000001},"min":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.435},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69199999999999995,"15min":0.73699999999999999},"min":{"1min":0.51200000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.008,"5min":1.177,"15min":2.133},"last":0.56699999999999995}]},{"osd":3,"last update":"Mon Sep 16 07:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.84899999999999998,"15min":0.84099999999999997},"min":{"1min":0.76700000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.0549999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.872,"15min":0.91200000000000003},"min":{"1min":0.64100000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0980000000000001,"5min":1.371,"15min":1.371},"last":0.628}]},{"osd":4,"last update":"Mon Sep 16 07:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.91500000000000004,"15min":0.92900000000000005},"min":{"1min":0.70799999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.044,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88200000000000001,"15min":0.91000000000000003},"min":{"1min":0.72299999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0740000000000001,"5min":1.484,"15min":1.484},"last":0.77300000000000002}]},{"osd":5,"last update":"Mon Sep 16 07:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.90500000000000003,"15min":0.91900000000000004},"min":{"1min":0.66900000000000004,"5min":0.441,"15min":0.441},"max":{"1min":1.1160000000000001,"5min":1.393,"15min":1.393},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.97499999999999998,"15min":1.0309999999999999},"min":{"1min":0.75700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.105,"5min":2.028,"15min":2.028},"last":0.85799999999999998}]}]},{"osd":5,"up_from":11,"seq":47244640334,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":4747264,"data_stored":2093575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.75900000000000001,"15min":0.72799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1080000000000001,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72599999999999998,"15min":0.74399999999999999},"min":{"1min":0.55700000000000005,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91700000000000004,"5min":1.036,"15min":1.036},"last":0.69699999999999995}]},{"osd":1,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.83999999999999997,"15min":0.80900000000000005},"min":{"1min":0.54000000000000004,"5min":0.438,"15min":0.438},"max":{"1min":1.0600000000000001,"5min":1.254,"15min":1.254},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.86599999999999999,"15min":0.84799999999999998},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.02,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.88200000000000001}]},{"osd":2,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.84599999999999997,"15min":0.82699999999999996},"min":{"1min":0.60199999999999998,"5min":0.59099999999999997,"15min":0.56100000000000005},"max":{"1min":0.92600000000000005,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81899999999999995,"15min":0.82899999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.46899999999999997},"max":{"1min":1.0780000000000001,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.86299999999999999}]},{"osd":3,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.70999999999999996},"min":{"1min":0.437,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96999999999999997,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.76300000000000001,"15min":0.78300000000000003},"min":{"1min":0.46600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.125,"5min":1.165,"15min":1.165},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.81200000000000006,"15min":0.80500000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.107,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81000000000000005,"15min":0.81799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0600000000000001,"5min":2.569,"15min":2.569},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:25:06.771 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T07:18:54.434573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=18, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=25, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:25:06.771 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T07:18:54.434573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=18, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=25, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:25:06.771 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T07:18:54.434573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=18, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=25, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:25:06.771 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T07:18:54.434573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=18, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=25, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:25:06.771 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T07:18:54.434573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=18, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=25, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:25:06.771 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T07:18:54.434573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=18, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=25, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:25:06.771 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T07:18:54.434573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=18, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=25, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:25:06.772 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T07:18:54.434573+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=18, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=25, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:25:06.772 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T07:18:47.386541+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=18, tm_sec=47, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=7, tm_min=25, tm_sec=2, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T07:25:06.772 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T07:25:26.773 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T07:25:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient: get_monmap_and_config 2024-09-16T07:25:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient: build_initial_monmap 2024-09-16T07:25:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient: monmap: 2024-09-16T07:25:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:26.892223+0000 2024-09-16T07:25:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:26.892223+0000 2024-09-16T07:25:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:26.894 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:26.894 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:26.894 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:26.894 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:26.894 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:26.894 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:26.894 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:26.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 1 Processor -- start 2024-09-16T07:25:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 1 -- start start 2024-09-16T07:25:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient: init 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f971009aaa0 0x7f9710132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient(hunting): picked mon.noname-a con 0x7f971009aaa0 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f971009dbb0 0x7f971009dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient(hunting): picked mon.noname-b con 0x7f971009dbb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97100999e0 addr v1:172.21.15.143:6789/0 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f971009ae80 con 0x7f97100999e0 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9710073ca0 con 0x7f971009aaa0 2024-09-16T07:25:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971009bde0 con 0x7f971009dbb0 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f9715649700 10 monclient(hunting): authenticate will time out at 1247.379934s 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.889+0000 7f970ffff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f971009aaa0 0x7f9710132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f971009aaa0 0x7f9710132140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49404/0 (socket says 172.21.15.37:49404) 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 1 -- 172.21.15.37:0/3595364176 learned_addr learned my addr 172.21.15.37:0/3595364176 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 10 monclient(hunting): get_auth_request con 0x7f971009aaa0 auth_method 0 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9707fff700 1 --2- 172.21.15.37:0/3595364176 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f971009dbb0 0x7f971009dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9707fff700 10 monclient(hunting): get_auth_request con 0x7f971009dbb0 auth_method 0 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9707fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9707fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9707fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9707fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9707fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9707fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 10 monclient(hunting): handle_auth_done global_id 54261 payload 274 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 1 -- 172.21.15.37:0/3595364176 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f971009dbb0 msgr2=0x7f971009dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 1 --2- 172.21.15.37:0/3595364176 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f971009dbb0 0x7f971009dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 1 -- 172.21.15.37:0/3595364176 >> v1:172.21.15.143:6789/0 conn(0x7f97100999e0 legacy=0x7f9710099e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 10 monclient: _finish_hunting 0 2024-09-16T07:25:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 1 monclient: found mon.noname-a 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 1 -- 172.21.15.37:0/3595364176 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971009c0c0 con 0x7f971009aaa0 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970ffff700 1 --2- 172.21.15.37:0/3595364176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f971009aaa0 0x7f9710132140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9700009940 tx=0x7f9700009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e87eb6220bf3c02 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 1 -- 172.21.15.37:0/3595364176 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700011830 con 0x7f971009aaa0 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 10 monclient: dump: 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 10 monclient: _finish_auth 0 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 10 monclient: _check_auth_tickets 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 5 monclient: authenticate success, global_id 54261 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 1 -- 172.21.15.37:0/3595364176 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9700011990 con 0x7f971009aaa0 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:26.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 4 set_mon_vals no callback set 2024-09-16T07:25:26.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 1 -- 172.21.15.37:0/3595364176 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700011d20 con 0x7f971009aaa0 2024-09-16T07:25:26.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:26.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:26.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f970f7fe700 10 monclient: dump: 2024-09-16T07:25:26.904 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:26.904 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:26.904 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:26.904 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 10 monclient: shutdown 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 1 -- 172.21.15.37:0/3595364176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f971009aaa0 msgr2=0x7f9710132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 1 --2- 172.21.15.37:0/3595364176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f971009aaa0 0x7f9710132140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9700009940 tx=0x7f9700009c50 comp rx=0 tx=0).stop 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 1 -- 172.21.15.37:0/3595364176 shutdown_connections 2024-09-16T07:25:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 1 --2- 172.21.15.37:0/3595364176 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f971009dbb0 0x7f971009dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 1 --2- 172.21.15.37:0/3595364176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f971009aaa0 0x7f9710132140 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 1 -- 172.21.15.37:0/3595364176 >> 172.21.15.37:0/3595364176 conn(0x7f97100a3f60 msgr2=0x7f97100a4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 1 -- 172.21.15.37:0/3595364176 shutdown_connections 2024-09-16T07:25:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 1 -- 172.21.15.37:0/3595364176 wait complete. 2024-09-16T07:25:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 10 monclient: build_initial_monmap 2024-09-16T07:25:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.893+0000 7f9715649700 10 monclient: monmap: 2024-09-16T07:25:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 1 Processor -- start 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 1 -- start start 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient: init 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97100999e0 0x7f97101c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient(hunting): picked mon.noname-c con 0x7f97100999e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f971009aaa0 0x7f971012e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient(hunting): picked mon.noname-b con 0x7f971009aaa0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f971009dbb0 0x7f971012edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient(hunting): picked mon.noname-a con 0x7f971009dbb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97101c6370 con 0x7f971009dbb0 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97100a1030 con 0x7f971009aaa0 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971009d5a0 con 0x7f97100999e0 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient(hunting): authenticate will time out at 1247.385559s 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97100999e0 0x7f97101c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97100999e0 0x7f97101c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:43346/0 (socket says 172.21.15.37:43346) 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 1 -- 172.21.15.37:0/699484604 learned_addr learned my addr 172.21.15.37:0/699484604 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 10 monclient(hunting): get_auth_request con 0x7f97100999e0 auth_method 0 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9714e48700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f971009dbb0 0x7f971012edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9714e48700 10 monclient(hunting): get_auth_request con 0x7f971009dbb0 auth_method 0 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9714e48700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9714e48700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9714e48700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9707fff700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f971009aaa0 0x7f971012e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9707fff700 10 monclient(hunting): get_auth_request con 0x7f971009aaa0 auth_method 0 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9707fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9707fff700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9707fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9714e48700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9714e48700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9714e48700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9707fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9707fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9707fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 10 monclient(hunting): handle_auth_done global_id 44624 payload 824 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 1 -- 172.21.15.37:0/699484604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f971009aaa0 msgr2=0x7f971012e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f971009aaa0 0x7f971012e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 1 -- 172.21.15.37:0/699484604 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f971009dbb0 msgr2=0x7f971012edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f971009dbb0 0x7f971012edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 10 monclient: _finish_hunting 0 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 1 monclient: found mon.noname-c 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 1 -- 172.21.15.37:0/699484604 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9700009570 con 0x7f97100999e0 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970ffff700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97100999e0 0x7f97101c5e30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f971006cc40 tx=0x7f9700004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 1 -- 172.21.15.37:0/699484604 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700004590 con 0x7f97100999e0 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 10 monclient: dump: 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 10 monclient: _finish_auth 0 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 10 monclient: _check_auth_tickets 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 5 monclient: authenticate success, global_id 44624 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 1 -- 172.21.15.37:0/699484604 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97000036c0 con 0x7f97100999e0 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 1 -- 172.21.15.37:0/699484604 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700003a70 con 0x7f97100999e0 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970effd700 4 set_mon_vals no callback set 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f970d7fa700 10 monclient: dump: 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:26.918 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient: _renew_subs 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 1 -- 172.21.15.37:0/699484604 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971011f280 con 0x7f97100999e0 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient: _renew_subs 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.897+0000 7f9715649700 1 -- 172.21.15.37:0/699484604 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971011ceb0 con 0x7f97100999e0 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.901+0000 7f9715649700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.901+0000 7f9715649700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.901+0000 7f9715649700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.901+0000 7f9715649700 1 -- 172.21.15.37:0/699484604 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9710132140 con 0x7f97100999e0 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.901+0000 7f970d7fa700 1 -- 172.21.15.37:0/699484604 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7f9700021070 con 0x7f97100999e0 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.901+0000 7f970d7fa700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f96f0035600 0x7f96f0037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.901+0000 7f970d7fa700 1 -- 172.21.15.37:0/699484604 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7f9700048f10 con 0x7f97100999e0 2024-09-16T07:25:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.901+0000 7f9707fff700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f96f0035600 0x7f96f0037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.901+0000 7f9707fff700 10 monclient: get_auth_request con 0x7f96f0035600 auth_method 0 2024-09-16T07:25:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.901+0000 7f9707fff700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f96f0035600 0x7f96f0037ac0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9710131550 tx=0x7f96f8009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.909+0000 7f970d7fa700 1 -- 172.21.15.37:0/699484604 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f970002d760 con 0x7f97100999e0 2024-09-16T07:25:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.909+0000 7f970d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:26.909+0000 7f970d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:27.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.077+0000 7f9715649700 1 -- 172.21.15.37:0/699484604 --> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97100a1260 con 0x7f96f0035600 2024-09-16T07:25:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.081+0000 7f970d7fa700 1 -- 172.21.15.37:0/699484604 <== mgr.4108 v2:172.21.15.143:6824/3705696220 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53571 (secure 0 0 0) 0x7f97100a1260 con 0x7f96f0035600 2024-09-16T07:25:27.088 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-16T07:25:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 -- 172.21.15.37:0/699484604 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f96f0035600 msgr2=0x7f96f0037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f96f0035600 0x7f96f0037ac0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9710131550 tx=0x7f96f8009240 comp rx=0 tx=0).stop 2024-09-16T07:25:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 10 monclient: shutdown 2024-09-16T07:25:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 -- 172.21.15.37:0/699484604 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97100999e0 msgr2=0x7f97101c5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97100999e0 0x7f97101c5e30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f971006cc40 tx=0x7f9700004770 comp rx=0 tx=0).stop 2024-09-16T07:25:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 -- 172.21.15.37:0/699484604 shutdown_connections 2024-09-16T07:25:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f971009dbb0 0x7f971012edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f971009aaa0 0x7f971012e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7f96f0035600 0x7f96f0037ac0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 --2- 172.21.15.37:0/699484604 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97100999e0 0x7f97101c5e30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 -- 172.21.15.37:0/699484604 >> 172.21.15.37:0/699484604 conn(0x7f97100a3f60 msgr2=0x7f9710130310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 -- 172.21.15.37:0/699484604 shutdown_connections 2024-09-16T07:25:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.085+0000 7f9715649700 1 -- 172.21.15.37:0/699484604 wait complete. 2024-09-16T07:25:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-09-16T07:25:27.102 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-16T07:25:26.561703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":196032,"kb_used_data":25744,"kb_used_omap":9,"kb_used_meta":170230,"kb_avail":566035008,"statfs":{"total":579820584960,"available":579619848192,"internally_reserved":0,"allocated":26361856,"data_stored":10445776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":174316198},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003393"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":295,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:25:09.407120+0000","last_change":"2024-09-16T07:25:09.407120+0000","last_active":"2024-09-16T07:25:09.407120+0000","last_peered":"2024-09-16T07:25:09.407120+0000","last_clean":"2024-09-16T07:25:09.407120+0000","last_became_active":"2024-09-16T07:18:55.463443+0000","last_became_peered":"2024-09-16T07:18:55.463443+0000","last_unstale":"2024-09-16T07:25:09.407120+0000","last_undegraded":"2024-09-16T07:25:09.407120+0000","last_fullsized":"2024-09-16T07:25:09.407120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:25:09.406846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:25:09.406846+0000","last_clean_scrub_stamp":"2024-09-16T07:25:09.406846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:35:06.031323+0000","scrub_duration":0.0028077670000000001,"objects_trimmed":0,"snaptrim_duration":0.00059939100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":299,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:25:06.630159+0000","last_change":"2024-09-16T07:25:06.630159+0000","last_active":"2024-09-16T07:25:06.630159+0000","last_peered":"2024-09-16T07:25:06.630159+0000","last_clean":"2024-09-16T07:25:06.630159+0000","last_became_active":"2024-09-16T07:18:55.467864+0000","last_became_peered":"2024-09-16T07:18:55.467864+0000","last_unstale":"2024-09-16T07:25:06.630159+0000","last_undegraded":"2024-09-16T07:25:06.630159+0000","last_fullsized":"2024-09-16T07:25:06.630159+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:25:06.629842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:25:06.629842+0000","last_clean_scrub_stamp":"2024-09-16T07:25:06.629842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:18:51.456169+0000","scrub_duration":0.0029973209999999998,"objects_trimmed":0,"snaptrim_duration":0.00066039799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":295,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:25:15.479661+0000","last_change":"2024-09-16T07:25:15.479876+0000","last_active":"2024-09-16T07:25:15.479661+0000","last_peered":"2024-09-16T07:25:15.479661+0000","last_clean":"2024-09-16T07:25:15.479661+0000","last_became_active":"2024-09-16T07:18:55.471978+0000","last_became_peered":"2024-09-16T07:18:55.471978+0000","last_unstale":"2024-09-16T07:25:15.479661+0000","last_undegraded":"2024-09-16T07:25:15.479661+0000","last_fullsized":"2024-09-16T07:25:15.479661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:25:15.479613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:25:15.479613+0000","last_clean_scrub_stamp":"2024-09-16T07:25:15.479613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:27:03.749441+0000","scrub_duration":0.0035370710000000001,"objects_trimmed":0,"snaptrim_duration":0.00077051799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":296,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:25:09.259293+0000","last_change":"2024-09-16T07:25:09.259293+0000","last_active":"2024-09-16T07:25:09.259293+0000","last_peered":"2024-09-16T07:25:09.259293+0000","last_clean":"2024-09-16T07:25:09.259293+0000","last_became_active":"2024-09-16T07:18:55.468584+0000","last_became_peered":"2024-09-16T07:18:55.468584+0000","last_unstale":"2024-09-16T07:25:09.259293+0000","last_undegraded":"2024-09-16T07:25:09.259293+0000","last_fullsized":"2024-09-16T07:25:09.259293+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:25:09.259012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:25:09.259012+0000","last_clean_scrub_stamp":"2024-09-16T07:25:09.259012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:45:04.126189+0000","scrub_duration":0.0029364389999999999,"objects_trimmed":0,"snaptrim_duration":0.000689417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"315'3","reported_seq":305,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:25:06.456541+0000","last_change":"2024-09-16T07:25:06.456541+0000","last_active":"2024-09-16T07:25:06.456541+0000","last_peered":"2024-09-16T07:25:06.456541+0000","last_clean":"2024-09-16T07:25:06.456541+0000","last_became_active":"2024-09-16T07:18:55.472671+0000","last_became_peered":"2024-09-16T07:18:55.472671+0000","last_unstale":"2024-09-16T07:25:06.456541+0000","last_undegraded":"2024-09-16T07:25:06.456541+0000","last_fullsized":"2024-09-16T07:25:06.456541+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'3","last_scrub_stamp":"2024-09-16T07:25:06.456318+0000","last_deep_scrub":"315'3","last_deep_scrub_stamp":"2024-09-16T07:25:06.456318+0000","last_clean_scrub_stamp":"2024-09-16T07:25:06.456318+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:02:12.710510+0000","scrub_duration":0.0065608339999999998,"objects_trimmed":0,"snaptrim_duration":0.000840762,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":311,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:25:06.159091+0000","last_change":"2024-09-16T07:25:06.159091+0000","last_active":"2024-09-16T07:25:06.159091+0000","last_peered":"2024-09-16T07:25:06.159091+0000","last_clean":"2024-09-16T07:25:06.159091+0000","last_became_active":"2024-09-16T07:18:55.469151+0000","last_became_peered":"2024-09-16T07:18:55.469151+0000","last_unstale":"2024-09-16T07:25:06.159091+0000","last_undegraded":"2024-09-16T07:25:06.159091+0000","last_fullsized":"2024-09-16T07:25:06.159091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:25:06.158821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:25:06.158821+0000","last_clean_scrub_stamp":"2024-09-16T07:25:06.158821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:36.364752+0000","scrub_duration":0.0027475469999999999,"objects_trimmed":0,"snaptrim_duration":0.00064042899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":314,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:25:10.758805+0000","last_change":"2024-09-16T07:25:10.759039+0000","last_active":"2024-09-16T07:25:10.758805+0000","last_peered":"2024-09-16T07:25:10.758805+0000","last_clean":"2024-09-16T07:25:10.758805+0000","last_became_active":"2024-09-16T07:18:55.469565+0000","last_became_peered":"2024-09-16T07:18:55.469565+0000","last_unstale":"2024-09-16T07:25:10.758805+0000","last_undegraded":"2024-09-16T07:25:10.758805+0000","last_fullsized":"2024-09-16T07:25:10.758805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T07:25:10.758758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T07:25:10.758758+0000","last_clean_scrub_stamp":"2024-09-16T07:25:10.758758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:17.949766+0000","scrub_duration":0.0031734979999999999,"objects_trimmed":0,"snaptrim_duration":0.00077454399999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"315'2","reported_seq":297,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:25:08.445918+0000","last_change":"2024-09-16T07:25:08.445918+0000","last_active":"2024-09-16T07:25:08.445918+0000","last_peered":"2024-09-16T07:25:08.445918+0000","last_clean":"2024-09-16T07:25:08.445918+0000","last_became_active":"2024-09-16T07:18:55.464362+0000","last_became_peered":"2024-09-16T07:18:55.464362+0000","last_unstale":"2024-09-16T07:25:08.445918+0000","last_undegraded":"2024-09-16T07:25:08.445918+0000","last_fullsized":"2024-09-16T07:25:08.445918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'2","last_scrub_stamp":"2024-09-16T07:25:08.445675+0000","last_deep_scrub":"315'2","last_deep_scrub_stamp":"2024-09-16T07:25:08.445675+0000","last_clean_scrub_stamp":"2024-09-16T07:25:08.445675+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:57:28.938291+0000","scrub_duration":0.0058563540000000002,"objects_trimmed":0,"snaptrim_duration":0.001212957,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"315'119","reported_seq":445,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-16T07:25:06.518265+0000","last_change":"2024-09-16T07:25:06.518416+0000","last_active":"2024-09-16T07:25:06.518265+0000","last_peered":"2024-09-16T07:25:06.518265+0000","last_clean":"2024-09-16T07:25:06.518265+0000","last_became_active":"2024-09-16T07:18:49.373551+0000","last_became_peered":"2024-09-16T07:18:49.373551+0000","last_unstale":"2024-09-16T07:25:06.518265+0000","last_undegraded":"2024-09-16T07:25:06.518265+0000","last_fullsized":"2024-09-16T07:25:06.518265+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'119","last_scrub_stamp":"2024-09-16T07:25:06.518236+0000","last_deep_scrub":"315'119","last_deep_scrub_stamp":"2024-09-16T07:25:06.518236+0000","last_clean_scrub_stamp":"2024-09-16T07:25:06.518236+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:07.601687+0000","scrub_duration":0.010738526999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":13,"seq":55834574931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35228,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94336612,"statfs":{"total":96636764160,"available":96600690688,"internally_reserved":0,"allocated":4214784,"data_stored":1564647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.78800000000000003,"15min":0.83699999999999997},"min":{"1min":0.496,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1040000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.78800000000000003,"15min":0.80800000000000005},"min":{"1min":0.54400000000000004,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.98799999999999999,"5min":1.329,"15min":1.329},"last":0.88900000000000001}]},{"osd":1,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.83299999999999996,"15min":0.82299999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.1779999999999999,"5min":2.177,"15min":2.177},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82599999999999996,"15min":0.872},"min":{"1min":0.58899999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.079,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.71599999999999997}]},{"osd":2,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.82699999999999996,"15min":0.82499999999999996},"min":{"1min":0.58899999999999997,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.091,"5min":1.23,"15min":1.23},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.92600000000000005,"15min":0.89300000000000002},"min":{"1min":0.60399999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.028,"5min":3.8610000000000002,"15min":3.8610000000000002},"last":0.96699999999999997}]},{"osd":4,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.73699999999999999,"15min":0.72799999999999998},"min":{"1min":0.47299999999999998,"5min":0.377,"15min":0.377},"max":{"1min":3.879,"5min":3.879,"15min":3.879},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.76900000000000002,"15min":0.78900000000000003},"min":{"1min":0.54900000000000004,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.7999999999999998,"5min":2.7999999999999998,"15min":2.7999999999999998},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.74299999999999999,"15min":0.751},"min":{"1min":0.57499999999999996,"5min":0.432,"15min":0.432},"max":{"1min":1.05,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.91100000000000003,"15min":0.86699999999999999},"min":{"1min":0.69099999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.171,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.01}]}]},{"osd":2,"up_from":13,"seq":55834574931,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31132,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340708,"statfs":{"total":96636764160,"available":96604884992,"internally_reserved":0,"allocated":4214784,"data_stored":1564647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58099999999999996,"15min":0.57599999999999996},"min":{"1min":0.44700000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.145,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.65000000000000002,"15min":0.629},"min":{"1min":0.505,"5min":0.316,"15min":0.316},"max":{"1min":0.79800000000000004,"5min":1.135,"15min":1.135},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Sep 16 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.628,"15min":0.64800000000000002},"min":{"1min":0.52100000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.79800000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.747,"15min":0.69699999999999995},"min":{"1min":0.49099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.74399999999999999}]},{"osd":3,"last update":"Mon Sep 16 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.79300000000000004,"15min":0.77400000000000002},"min":{"1min":0.57899999999999996,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.111,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.89800000000000002,"15min":0.86299999999999999},"min":{"1min":0.72599999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.242,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.86699999999999999}]},{"osd":4,"last update":"Mon Sep 16 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.92000000000000004,"15min":0.91300000000000003},"min":{"1min":0.67300000000000004,"5min":0.496,"15min":0.496},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84599999999999997,"15min":0.89400000000000002},"min":{"1min":0.63800000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.95999999999999996,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Sep 16 07:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90400000000000003,"15min":0.91600000000000004},"min":{"1min":0.72499999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.177,"5min":1.419,"15min":1.419},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91100000000000003,"15min":0.93700000000000006},"min":{"1min":0.70199999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.284,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.80800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574931,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35232,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94336608,"statfs":{"total":96636764160,"available":96600686592,"internally_reserved":0,"allocated":4218880,"data_stored":1564666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 07:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65200000000000002,"15min":0.65600000000000003},"min":{"1min":0.48999999999999999,"5min":0.42099999999999999,"15min":0.41899999999999998},"max":{"1min":0.90900000000000003,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Sep 16 07:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58399999999999996,"15min":0.626},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81799999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.70299999999999996,"15min":0.75},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.43099999999999999},"max":{"1min":0.86899999999999999,"5min":1.0720000000000001,"15min":1.2829999999999999},"last":0.68600000000000005}]},{"osd":3,"last update":"Mon Sep 16 07:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.85199999999999998,"15min":0.86199999999999999},"min":{"1min":0.60299999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.0669999999999999,"5min":1.181,"15min":1.611},"last":0.998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.84999999999999998,"15min":0.85399999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.95299999999999996,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.77000000000000002}]},{"osd":4,"last update":"Mon Sep 16 07:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.874,"15min":0.91100000000000003},"min":{"1min":0.76700000000000002,"5min":0.621,"15min":0.621},"max":{"1min":1.115,"5min":1.472,"15min":1.998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.91300000000000003,"15min":0.91500000000000004},"min":{"1min":0.70999999999999996,"5min":0.63,"15min":0.63},"max":{"1min":1.2430000000000001,"5min":1.3080000000000001,"15min":1.4019999999999999},"last":0.86799999999999999}]},{"osd":5,"last update":"Mon Sep 16 07:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.89000000000000001,"15min":0.97799999999999998},"min":{"1min":0.66500000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2010000000000001,"5min":1.3420000000000001,"15min":1.673},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.97199999999999998,"15min":0.99199999999999999},"min":{"1min":0.68600000000000005,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.151,"5min":1.387,"15min":1.825},"last":1.133}]}]},{"osd":4,"up_from":12,"seq":51539607635,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31136,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340704,"statfs":{"total":96636764160,"available":96604880896,"internally_reserved":0,"allocated":4218880,"data_stored":1564666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.82699999999999996,"15min":0.84699999999999998},"min":{"1min":0.66900000000000004,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1060000000000001,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.84699999999999998,"15min":0.82599999999999996},"min":{"1min":0.70599999999999996,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2270000000000001,"5min":1.272,"15min":1.272},"last":0.78000000000000003}]},{"osd":1,"last update":"Mon Sep 16 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.80000000000000004,"15min":0.81799999999999995},"min":{"1min":0.60299999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.089,"5min":1.462,"15min":1.462},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.87,"5min":0.83399999999999996,"15min":0.86199999999999999},"min":{"1min":0.58599999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.268,"5min":1.391,"15min":1.548},"last":0.72499999999999998}]},{"osd":2,"last update":"Mon Sep 16 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.82199999999999995,"15min":0.78200000000000003},"min":{"1min":0.52200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.1819999999999999,"5min":2.246,"15min":2.246},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.86399999999999999,"15min":0.88800000000000001},"min":{"1min":0.66700000000000004,"5min":0.435,"15min":0.435},"max":{"1min":1.1759999999999999,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.89100000000000001}]},{"osd":3,"last update":"Mon Sep 16 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.72799999999999998,"15min":0.80800000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85499999999999998,"5min":2.423,"15min":2.423},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.76500000000000001,"15min":0.78500000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95199999999999996,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.74099999999999999}]},{"osd":5,"last update":"Mon Sep 16 07:24:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80400000000000005,"15min":0.82599999999999996},"min":{"1min":0.47699999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.3740000000000001,"5min":1.415,"15min":1.8260000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.82999999999999996,"15min":0.86899999999999999},"min":{"1min":0.55400000000000005,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.1080000000000001,"5min":1.284,"15min":1.776},"last":0.67000000000000004}]}]},{"osd":1,"up_from":12,"seq":51539607634,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":4747264,"data_stored":2093575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58199999999999996,"15min":0.57699999999999996},"min":{"1min":0.46500000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.97299999999999998,"5min":1.107,"15min":1.107},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.63100000000000001,"15min":0.66300000000000003},"min":{"1min":0.51600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":1.23},"last":0.66100000000000003}]},{"osd":2,"last update":"Mon Sep 16 07:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.66400000000000003,"15min":0.64100000000000001},"min":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69199999999999995,"15min":0.73699999999999999},"min":{"1min":0.51200000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.008,"5min":1.177,"15min":2.133},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Sep 16 07:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.84899999999999998,"15min":0.84099999999999997},"min":{"1min":0.76700000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.0549999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.872,"15min":0.91200000000000003},"min":{"1min":0.64100000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0980000000000001,"5min":1.371,"15min":1.371},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon Sep 16 07:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.91500000000000004,"15min":0.92900000000000005},"min":{"1min":0.70799999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.044,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88200000000000001,"15min":0.91000000000000003},"min":{"1min":0.72299999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.0740000000000001,"5min":1.484,"15min":1.484},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon Sep 16 07:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.90500000000000003,"15min":0.91900000000000004},"min":{"1min":0.66900000000000004,"5min":0.441,"15min":0.441},"max":{"1min":1.1160000000000001,"5min":1.393,"15min":1.393},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.97499999999999998,"15min":1.0309999999999999},"min":{"1min":0.75700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.105,"5min":2.028,"15min":2.028},"last":0.95599999999999996}]}]},{"osd":5,"up_from":11,"seq":47244640338,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31652,"kb_used_data":4636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340188,"statfs":{"total":96636764160,"available":96604352512,"internally_reserved":0,"allocated":4747264,"data_stored":2093575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.75900000000000001,"15min":0.72799999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.1080000000000001,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72599999999999998,"15min":0.74399999999999999},"min":{"1min":0.55700000000000005,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91700000000000004,"5min":1.036,"15min":1.036},"last":0.92700000000000005}]},{"osd":1,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.83999999999999997,"15min":0.80900000000000005},"min":{"1min":0.54000000000000004,"5min":0.438,"15min":0.438},"max":{"1min":1.0600000000000001,"5min":1.254,"15min":1.254},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.86599999999999999,"15min":0.84799999999999998},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.02,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.82299999999999995}]},{"osd":2,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.84599999999999997,"15min":0.82699999999999996},"min":{"1min":0.60199999999999998,"5min":0.59099999999999997,"15min":0.56100000000000005},"max":{"1min":0.92600000000000005,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.755},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.81899999999999995,"15min":0.82899999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.46899999999999997},"max":{"1min":1.0780000000000001,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.79100000000000004}]},{"osd":3,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.70999999999999996},"min":{"1min":0.437,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.96999999999999997,"5min":2.5099999999999998,"15min":2.5099999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.76300000000000001,"15min":0.78300000000000003},"min":{"1min":0.46600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.125,"5min":1.165,"15min":1.165},"last":0.90400000000000003}]},{"osd":4,"last update":"Mon Sep 16 07:24:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.81200000000000006,"15min":0.80500000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.107,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.81000000000000005,"15min":0.81799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0600000000000001,"5min":2.569,"15min":2.569},"last":0.89100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T07:25:27.104 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T07:25:27.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.209+0000 7fb35759e700 10 monclient: get_monmap_and_config 2024-09-16T07:25:27.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.209+0000 7fb35759e700 10 monclient: build_initial_monmap 2024-09-16T07:25:27.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.209+0000 7fb35759e700 10 monclient: monmap: 2024-09-16T07:25:27.215 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:27.215 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:27.215 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:25:27.214395+0000 2024-09-16T07:25:27.215 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:25:27.214395+0000 2024-09-16T07:25:27.216 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:27.216 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:27.216 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-09-16T07:25:27.216 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.143:6789/0 mon.noname-c-legacy 2024-09-16T07:25:27.216 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-09-16T07:25:27.216 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-09-16T07:25:27.216 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.143:3300/0 mon.noname-c 2024-09-16T07:25:27.217 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:27.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.209+0000 7fb35759e700 1 Processor -- start 2024-09-16T07:25:27.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.209+0000 7fb35759e700 1 -- start start 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.209+0000 7fb35759e700 10 monclient: init 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb35806a750 addr v1:172.21.15.143:6789/0 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb35806d8f0 0x7fb35813a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient(hunting): picked mon.noname-a con 0x7fb35806d8f0 addr v2:172.21.15.37:3300/0 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb35806b240 0x7fb35813cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient(hunting): picked mon.noname-c con 0x7fb35806b240 addr v2:172.21.15.143:3300/0 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35806d460 con 0x7fb35806a750 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:27.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35809ec20 con 0x7fb35806d8f0 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35809ef00 con 0x7fb35806b240 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient(hunting): authenticate will time out at 1247.700495s 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb35806d8f0 0x7fb35813a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb35806d8f0 0x7fb35813a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49420/0 (socket says 172.21.15.37:49420) 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 1 -- 172.21.15.37:0/2600559738 learned_addr learned my addr 172.21.15.37:0/2600559738 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 10 monclient(hunting): get_auth_request con 0x7fb35806d8f0 auth_method 0 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:27.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb356d9d700 1 --2- 172.21.15.37:0/2600559738 >> v2:172.21.15.143:3300/0 conn(0x7fb35806b240 0x7fb35813cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb356d9d700 10 monclient(hunting): get_auth_request con 0x7fb35806b240 auth_method 0 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb356d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb356d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb356d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 10 monclient(hunting): handle_auth_done global_id 54267 payload 274 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 1 -- 172.21.15.37:0/2600559738 >> v2:172.21.15.143:3300/0 conn(0x7fb35806b240 msgr2=0x7fb35813cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 1 --2- 172.21.15.37:0/2600559738 >> v2:172.21.15.143:3300/0 conn(0x7fb35806b240 0x7fb35813cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 1 -- 172.21.15.37:0/2600559738 >> v1:172.21.15.143:6789/0 conn(0x7fb35806a750 legacy=0x7fb35806ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 10 monclient: _finish_hunting 0 2024-09-16T07:25:27.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 1 monclient: found mon.noname-a 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 1 -- 172.21.15.37:0/2600559738 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35809e940 con 0x7fb35806d8f0 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb356d9d700 1 --2- 172.21.15.37:0/2600559738 >> v2:172.21.15.143:3300/0 conn(0x7fb35806b240 0x7fb35813cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb355d9b700 1 --2- 172.21.15.37:0/2600559738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806d8f0 0x7fb35813a670 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb3580a1440 tx=0x7fb348009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827647533cb2685b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 1 -- 172.21.15.37:0/2600559738 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34800b8a0 con 0x7fb35806d8f0 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 10 monclient: dump: 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:27.227 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 10 monclient: _finish_auth 0 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 10 monclient: _check_auth_tickets 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 5 monclient: authenticate success, global_id 54267 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 1 -- 172.21.15.37:0/2600559738 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34800ba00 con 0x7fb35806d8f0 2024-09-16T07:25:27.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient: get_monmap_and_config success 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 4 set_mon_vals no callback set 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 1 -- 172.21.15.37:0/2600559738 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34800bd90 con 0x7fb35806d8f0 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35559a700 10 monclient: dump: 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:27.229 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient: shutdown 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 -- 172.21.15.37:0/2600559738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806d8f0 msgr2=0x7fb35813a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 --2- 172.21.15.37:0/2600559738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806d8f0 0x7fb35813a670 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb3580a1440 tx=0x7fb348009bf0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 -- 172.21.15.37:0/2600559738 shutdown_connections 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 --2- 172.21.15.37:0/2600559738 >> v2:172.21.15.143:3300/0 conn(0x7fb35806b240 0x7fb35813cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 --2- 172.21.15.37:0/2600559738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806d8f0 0x7fb35813a670 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 -- 172.21.15.37:0/2600559738 >> 172.21.15.37:0/2600559738 conn(0x7fb3580a1530 msgr2=0x7fb3580a1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 -- 172.21.15.37:0/2600559738 shutdown_connections 2024-09-16T07:25:27.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 -- 172.21.15.37:0/2600559738 wait complete. 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient: build_initial_monmap 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 10 monclient: monmap: 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.noname-b 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:27.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.213+0000 7fb35759e700 1 Processor -- start 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 1 -- start start 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient: init 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient: _reopen_session rank -1 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806a750 0x7fb358134e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient(hunting): picked mon.noname-a con 0x7fb35806a750 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb35806b240 0x7fb358135370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient(hunting): picked mon.noname-c con 0x7fb35806b240 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb35806d8f0 0x7fb35811bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient(hunting): picked mon.noname-b con 0x7fb35806d8f0 addr [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35811e550 con 0x7fb35806a750 2024-09-16T07:25:27.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35811fc50 con 0x7fb35806d8f0 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient(hunting): start opening mon connection 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35806e070 con 0x7fb35806b240 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient(hunting): _renew_subs 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient(hunting): authenticate will time out at 1247.704581s 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806a750 0x7fb358134e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb355d9b700 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb35806b240 0x7fb358135370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806a750 0x7fb358134e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49422/0 (socket says 172.21.15.37:49422) 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 1 -- 172.21.15.37:0/349756309 learned_addr learned my addr 172.21.15.37:0/349756309 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 10 monclient(hunting): get_auth_request con 0x7fb35806a750 auth_method 0 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:27.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb355d9b700 10 monclient(hunting): get_auth_request con 0x7fb35806b240 auth_method 0 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb355d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb355d9b700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb355d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb356d9d700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb35806d8f0 0x7fb35811bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb356d9d700 10 monclient(hunting): get_auth_request con 0x7fb35806d8f0 auth_method 0 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb356d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb356d9d700 10 monclient(hunting): _init_auth method 2 2024-09-16T07:25:27.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb356d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-16T07:25:27.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb355d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:27.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb355d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:27.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb355d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:27.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb356d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T07:25:27.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb356d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T07:25:27.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb356d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 10 monclient(hunting): handle_auth_done global_id 54270 payload 824 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 1 -- 172.21.15.37:0/349756309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb35806b240 msgr2=0x7fb358135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb35806b240 0x7fb358135370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 1 -- 172.21.15.37:0/349756309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb35806d8f0 msgr2=0x7fb35811bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb35806d8f0 0x7fb35811bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 10 monclient: _finish_hunting 0 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 1 monclient: found mon.noname-a 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 1 -- 172.21.15.37:0/349756309 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb348009570 con 0x7fb35806a750 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb355d9b700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb35806b240 0x7fb358135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35659c700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806a750 0x7fb358134e30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c500 tx=0x7fb34c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:27.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 1 -- 172.21.15.37:0/349756309 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0052d0 con 0x7fb35806a750 2024-09-16T07:25:27.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:27.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T07:25:27.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 10 monclient: dump: 2024-09-16T07:25:27.237 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:27.237 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:27.237 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:27.237 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:27.237 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 10 monclient: _finish_auth 0 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 10 monclient: _check_auth_tickets 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 5 monclient: authenticate success, global_id 54270 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient: _renew_subs 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 1 -- 172.21.15.37:0/349756309 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35813ce10 con 0x7fb35806a750 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 1 -- 172.21.15.37:0/349756309 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c005430 con 0x7fb35806a750 2024-09-16T07:25:27.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-16T07:25:27.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 1 -- 172.21.15.37:0/349756309 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0055a0 con 0x7fb35806a750 2024-09-16T07:25:27.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-16T07:25:27.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb354d99700 4 set_mon_vals no callback set 2024-09-16T07:25:27.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T07:25:27.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 10 monclient: dump: 2024-09-16T07:25:27.240 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-09-16T07:25:27.240 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 2b3c1781-6e4a-42f3-a6cb-6f1008258949 2024-09-16T07:25:27.240 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:27.240 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-09-16T07:18:06.498847+0000 2024-09-16T07:25:27.240 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 18 (reef) 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient: _renew_subs 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb35759e700 1 -- 172.21.15.37:0/349756309 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35811e240 con 0x7fb35806a750 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.217+0000 7fb346ffd700 1 -- 172.21.15.37:0/349756309 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) v1 ==== 43307+0+0 (secure 0 0 0) 0x7fb34c0036a0 con 0x7fb35806a750 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.221+0000 7fb35759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.221+0000 7fb35759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.221+0000 7fb35759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.221+0000 7fb35759e700 1 -- 172.21.15.37:0/349756309 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb358067890 con 0x7fb35806a750 2024-09-16T07:25:27.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.221+0000 7fb346ffd700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb338035600 0x7fb338037ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T07:25:27.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.221+0000 7fb346ffd700 1 -- 172.21.15.37:0/349756309 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 4346+0+0 (secure 0 0 0) 0x7fb34c046c50 con 0x7fb35806a750 2024-09-16T07:25:27.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.221+0000 7fb355d9b700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb338035600 0x7fb338037ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T07:25:27.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.221+0000 7fb355d9b700 10 monclient: get_auth_request con 0x7fb338035600 auth_method 0 2024-09-16T07:25:27.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.221+0000 7fb355d9b700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb338035600 0x7fb338037ac0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb348000c00 tx=0x7fb348008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T07:25:27.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.229+0000 7fb346ffd700 1 -- 172.21.15.37:0/349756309 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb34c010b10 con 0x7fb35806a750 2024-09-16T07:25:27.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.229+0000 7fb346ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T07:25:27.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.229+0000 7fb346ffd700 10 monclient: _finish_command 1 = system:0 2024-09-16T07:25:27.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.397+0000 7fb35759e700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T07:25:27.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.397+0000 7fb35759e700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T07:25:27.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.397+0000 7fb35759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-09-16T07:25:27.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.397+0000 7fb35759e700 1 -- 172.21.15.37:0/349756309 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fb35806ab30 con 0x7fb35806a750 2024-09-16T07:25:27.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.405+0000 7fb346ffd700 1 -- 172.21.15.37:0/349756309 <== mon.0 v2:172.21.15.37:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb34c04c020 con 0x7fb35806a750 2024-09-16T07:25:27.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.405+0000 7fb346ffd700 10 monclient: handle_config config(1 keys) v1 2024-09-16T07:25:27.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.405+0000 7fb354d99700 4 set_mon_vals no callback set 2024-09-16T07:25:27.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.405+0000 7fb354d99700 10 set_mon_vals mon_health_to_clog = false 2024-09-16T07:25:27.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb346ffd700 1 -- 172.21.15.37:0/349756309 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) v1 ==== 123+0+0 (secure 0 0 0) 0x7fb34c010700 con 0x7fb35806a750 2024-09-16T07:25:27.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb346ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T07:25:27.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb346ffd700 10 monclient: _finish_command 2 = system:0 2024-09-16T07:25:27.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 -- 172.21.15.37:0/349756309 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb338035600 msgr2=0x7fb338037ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:27.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb338035600 0x7fb338037ac0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb348000c00 tx=0x7fb348008030 comp rx=0 tx=0).stop 2024-09-16T07:25:27.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 10 monclient: shutdown 2024-09-16T07:25:27.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T07:25:27.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 -- 172.21.15.37:0/349756309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806a750 msgr2=0x7fb358134e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T07:25:27.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806a750 0x7fb358134e30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c500 tx=0x7fb34c00c810 comp rx=0 tx=0).stop 2024-09-16T07:25:27.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 -- 172.21.15.37:0/349756309 shutdown_connections 2024-09-16T07:25:27.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb35806d8f0 0x7fb35811bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.143:6824/3705696220,v1:172.21.15.143:6825/3705696220] conn(0x7fb338035600 0x7fb338037ac0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb35806b240 0x7fb358135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 --2- 172.21.15.37:0/349756309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb35806a750 0x7fb358134e30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T07:25:27.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 -- 172.21.15.37:0/349756309 >> 172.21.15.37:0/349756309 conn(0x7fb3580a1530 msgr2=0x7fb35806fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T07:25:27.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 -- 172.21.15.37:0/349756309 shutdown_connections 2024-09-16T07:25:27.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-16T07:25:27.409+0000 7fb35759e700 1 -- 172.21.15.37:0/349756309 wait complete. 2024-09-16T07:25:27.433 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T07:25:27.433 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T07:25:27.433 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T07:25:27.433 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:25:27.514 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T07:25:27.514 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T07:25:27.514 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:25:27.606 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T07:25:27.606 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T07:25:27.607 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:25:27.699 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T07:25:27.699 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T07:25:27.699 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:25:29.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-16T07:25:33.801 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T07:25:33.802 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T07:25:33.802 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:25:34.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-16T07:25:34.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-16T07:25:39.904 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T07:25:39.905 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T07:25:39.905 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:25:39.980 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T07:25:39.980 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T07:25:39.980 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T07:25:39.981 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:25:40.013 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T07:25:40.013 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T07:25:40.013 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T07:25:40.013 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:25:40.065 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T07:25:40.066 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T07:25:40.066 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:25:40.113 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T07:25:40.114 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T07:25:40.114 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T07:25:40.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~6s 2024-09-16T07:25:40.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-09-16T07:25:40.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-09-16T07:25:40.157 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T07:25:40.158 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T07:25:40.158 DEBUG:teuthology.orchestra.run.smithi037:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | head -n 1 2024-09-16T07:25:40.180 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T07:25:40.181 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T07:25:40.459 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T07:25:40.459 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T07:25:40.684 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T07:25:40.684 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T07:25:40.898 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T07:25:40.898 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T07:25:41.167 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T07:25:41.168 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T07:25:41.457 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T07:25:41.458 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T07:25:41.633 INFO:tasks.ceph:Archiving mon data... 2024-09-16T07:25:41.634 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621/data/mon.a.tgz 2024-09-16T07:25:41.634 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-09-16T07:25:41.641 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.EEnqNU0G1U 2024-09-16T07:25:41.641 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.EEnqNU0G1U 2024-09-16T07:25:41.698 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.EEnqNU0G1U 2024-09-16T07:25:41.720 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.EEnqNU0G1U is 175KB 2024-09-16T07:25:41.736 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.EEnqNU0G1U 2024-09-16T07:25:41.741 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621/data/mon.c.tgz 2024-09-16T07:25:41.741 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-09-16T07:25:41.789 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.hx6BSymYGj 2024-09-16T07:25:41.790 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.hx6BSymYGj 2024-09-16T07:25:41.846 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.hx6BSymYGj 2024-09-16T07:25:41.868 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.hx6BSymYGj is 177KB 2024-09-16T07:25:41.921 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.hx6BSymYGj 2024-09-16T07:25:41.928 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621/data/mon.b.tgz 2024-09-16T07:25:41.928 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-09-16T07:25:41.934 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.BVyQ4jDRs8 2024-09-16T07:25:41.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.BVyQ4jDRs8 2024-09-16T07:25:41.990 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.BVyQ4jDRs8 2024-09-16T07:25:42.011 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.BVyQ4jDRs8 is 184KB 2024-09-16T07:25:42.025 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.BVyQ4jDRs8 2024-09-16T07:25:42.033 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T07:25:42.033 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T07:25:42.035 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T07:25:42.091 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T07:25:42.091 DEBUG:teuthology.orchestra.run.smithi037:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T07:25:42.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T07:25:42.114 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T07:25:42.121 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T07:25:42.128 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T07:25:42.129 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621/remote/smithi037/crash 2024-09-16T07:25:42.130 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T07:25:42.187 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621/remote/smithi143/crash 2024-09-16T07:25:42.188 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T07:25:42.205 INFO:tasks.ceph:Compressing logs... 2024-09-16T07:25:42.205 DEBUG:teuthology.orchestra.run.smithi037:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T07:25:42.208 DEBUG:teuthology.orchestra.run.smithi143:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T07:25:45.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-16T07:25:45.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-09-16T07:25:45.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-09-16T07:25:51.454 INFO:tasks.ceph:Archiving logs... 2024-09-16T07:25:51.455 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621/remote/smithi037/log 2024-09-16T07:25:51.456 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T07:25:52.677 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621/remote/smithi143/log 2024-09-16T07:25:52.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T07:25:54.001 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T07:25:54.014 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T07:25:54.015 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T07:25:54.017 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T07:25:54.085 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T07:25:54.097 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T07:25:54.098 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T07:25:54.100 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T07:25:54.119 INFO:teuthology.orchestra.run.smithi037.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T07:25:54.119 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================== 2024-09-16T07:25:54.120 INFO:teuthology.orchestra.run.smithi037.stdout:*hv01.front.sepi 67.205.162.81 3 u 62 64 377 0.073 -2.193 2.308 2024-09-16T07:25:54.120 INFO:teuthology.orchestra.run.smithi037.stdout:+hv02.front.sepi 63.231.80.2 3 u 64 64 377 0.109 1.212 3.597 2024-09-16T07:25:54.120 INFO:teuthology.orchestra.run.smithi037.stdout:+hv03.front.sepi 74.6.168.72 3 u 58 64 377 0.072 -1.751 2.255 2024-09-16T07:25:54.120 INFO:teuthology.orchestra.run.smithi037.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:25:54.123 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T07:25:54.123 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2024-09-16T07:25:54.123 INFO:teuthology.orchestra.run.smithi143.stdout:+hv01.front.sepi 67.205.162.81 3 u 58 64 377 0.092 -3.289 1.544 2024-09-16T07:25:54.124 INFO:teuthology.orchestra.run.smithi143.stdout:+hv02.front.sepi 63.231.80.2 3 u 62 64 377 0.096 3.071 1.528 2024-09-16T07:25:54.124 INFO:teuthology.orchestra.run.smithi143.stdout:*hv03.front.sepi 74.6.168.72 3 u 61 64 377 0.084 -2.831 1.580 2024-09-16T07:25:54.124 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T07:25:54.125 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T07:25:54.175 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T07:25:54.175 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T07:25:54.226 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T07:25:54.279 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T07:25:54.337 INFO:teuthology.task.internal:Duration was 936.676758 seconds 2024-09-16T07:25:54.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T07:25:54.386 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T07:25:54.386 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T07:25:54.388 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T07:25:54.439 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T07:25:54.440 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi037.front.sepia.ceph.com 2024-09-16T07:25:54.440 DEBUG:teuthology.orchestra.run.smithi037:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T07:25:54.453 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-09-16T07:25:54.454 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T07:25:54.470 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T07:25:54.470 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T07:25:54.472 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T07:25:54.510 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T07:25:54.510 DEBUG:teuthology.orchestra.run.smithi037:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T07:25:54.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T07:25:54.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T07:25:54.638 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T07:25:54.639 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T07:25:54.656 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T07:25:54.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T07:25:54.693 DEBUG:teuthology.orchestra.run.smithi037:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:25:54.702 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:25:54.714 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = core 2024-09-16T07:25:54.729 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-09-16T07:25:54.760 DEBUG:teuthology.orchestra.run.smithi037:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:25:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:25:54.767 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T07:25:54.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T07:25:54.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T07:25:54.782 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T07:25:54.783 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621/remote/smithi037 2024-09-16T07:25:54.783 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T07:25:54.809 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906621/remote/smithi143 2024-09-16T07:25:54.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T07:25:54.835 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T07:25:54.835 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T07:25:54.850 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T07:25:54.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T07:25:54.901 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T07:25:54.902 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T07:25:54.937 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T07:25:54.937 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T07:25:54.939 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T07:25:54.945 INFO:teuthology.orchestra.run.smithi037.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 07:25 /home/ubuntu/cephtest 2024-09-16T07:25:54.947 INFO:teuthology.orchestra.run.smithi143.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 07:25 /home/ubuntu/cephtest 2024-09-16T07:25:54.949 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T07:25:55.000 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T07:25:55.116 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} thrashers/sync-many workloads/pool-create-delete} duration: 936.6767578125 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T07:25:55.116 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T07:25:55.201 INFO:teuthology.run:pass